19:01:25 <DavidWHodgins> #startmeeting
19:01:25 <Inigo_Montoya> Meeting started Thu Apr  6 19:01:25 2017 UTC.  The chair is DavidWHodgins. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:25 <Inigo_Montoya> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:01:42 <DavidWHodgins> #chair wilcal
19:01:42 <Inigo_Montoya> Current chairs: DavidWHodgins wilcal
19:01:57 <DavidWHodgins> #topic * Who's new? - If you are then come and say hello.
19:02:29 <DavidWHodgins> Several nics I don't recognize. If you haven't been to a qa team irc meeting before, please introduce yourself
19:03:15 <DavidWHodgins> g0d355__: Care to introduce yourself?
19:03:29 <DavidWHodgins> jibz: ?
19:04:08 <wilcal> Nothing :-0
19:04:10 <DavidWHodgins> Just lurking I guess, which is fine
19:04:13 <jibz> Yes ?
19:04:17 <wilcal> hello lewys
19:04:26 <wilcal> feel free to lurk'
19:04:28 <lewyssmith> Hello Bill & everyone.
19:04:29 <DavidWHodgins> #chair lewyssmith
19:04:29 <Inigo_Montoya> Current chairs: DavidWHodgins lewyssmith wilcal
19:04:43 <wilcal> hello lewys
19:04:47 <DavidWHodgins> jibz: Are you new here? I don't remember your nic off hand
19:04:55 <jibz> Hello everyone, whats happend ?
19:05:00 <jibz> Oh ha
19:05:28 <jibz> Yes... A little bit new. I'm just curious, so I came in all mageias room.
19:05:57 <wilcal> Great way to see what's going on
19:06:02 <DavidWHodgins> Well, this is the qa team irc meeting. Interested in joining the qa team?
19:06:03 <jibz> I'm watching here for... two weeks already... I guess.
19:06:10 <lewyssmith> jibz:  where are you?
19:06:48 <DavidWHodgins> I'm Dave Hodgins, leader of the qa team, from London, Ontario, Canada. Been with Mageia since the first alpha of Mageia 1.
19:06:52 <jibz> Hum... No, I can't, every thing I saw on this IRC, I didn't understand the words.
19:07:27 <lewyssmith> Evening Len.
19:07:33 <DavidWHodgins> Ok. Feel free to watch the channel, and if you like, subscribe to the mailing list qa-discuss at ml.mageia.org
19:07:36 <wilcal> "wilcal" for (Wil)liam in (Cal)ifornia, live near San Diego ( Surf City USA )
19:07:57 <DavidWHodgins> Let's move on then.
19:08:02 <DavidWHodgins> #topic * Mageia 6 - Testing M6
19:08:18 <DavidWHodgins> Let's start with, any new problems found?
19:08:29 <jibz> I'm for now in germany. I'm on mageia for 4 months. But I already was few years ago.
19:08:38 <wilcal> I'm not finding any new big wrinkles
19:09:00 <DavidWHodgins> All reports I've seen so far have been good
19:09:03 <lewyssmith> We are still bogged down with EFI/USB not booting for the Classic. This is an old issue which we should not still have.
19:09:22 <DavidWHodgins> Ouch
19:09:28 <tarazed> Yes it is frustrating
19:09:29 <wilcal> Nvidia issue seems to be behind us
19:09:43 <DavidWHodgins> That's good. That was a very serious one.
19:09:55 <DavidWHodgins> So is the efi/usb though
19:10:42 <wilcal> EFI thingy keeps coming back and biting us in the behind
19:11:16 <DavidWHodgins> EFI is a pain. Unlike bios, each manufacturer has their own "standards".
19:11:34 <bri_an> yeah - not fun
19:11:55 <lewyssmith> It is *not* a pain, Dave. Our problems have nothing to do with manufacturere conformance.
19:12:07 <wilcal> so is that the only thing holding up release of the pretest iso's
19:12:32 <DavidWHodgins> I think so. With bugzilla down, hard to confirm
19:12:59 <wilcal> all that's down because of the server upgrades right?
19:13:02 <marja> were all needed pre-tests done? I have the impression from the pretesting pads that more tests are needed for the most recent isos
19:13:06 <DavidWHodgins> wilcal: Yes
19:13:25 <DavidWHodgins> marja: Yes, more tests are needed
19:14:26 <wilcal> I don't think I found anything in the recent batch that would hold up the pretest isos but I'm not testing the EFI thing
19:14:39 <marja> wilcal: I don't think we need bugzilla to know whether the pre-test isos can be released to QA for further testing
19:14:49 <wilcal> that's correct
19:15:17 <DavidWHodgins> We just need to know whether the latest pretesting iso images fix the efi/usb booting
19:15:23 <wilcal> the last thing I had an issue with is the LiveDVD's wouldn't install GRUB2 and the latest batch fixed that
19:16:16 <DavidWHodgins> If we can confirm that, then we will release the pretesting iso images for wider qa testing
19:16:27 <DavidWHodgins> If not, we have to get that fixed.
19:16:29 <tarazed> The message was not real; erroneous
19:16:36 <wilcal> IMO we're ready for that
19:17:06 <DavidWHodgins> tarazed: Which message?
19:17:21 <marja> I didn't see erroneous messages in my non-EFI installs from the Live isos
19:17:25 <tarazed> The "cannot install grub2"
19:17:26 <lewyssmith> I see several complaints about the 64-bit Classic and Plasma -> IceWM.
19:17:30 <marja> with the current isos
19:18:07 <DavidWHodgins> That type of issue can be sorted after the iso images are released for wider qa testing
19:18:58 <marja> I think all 32bit isos are fine for further testing, only the CI wasn't tested in VBox
19:19:11 <tarazed> marja: not in this round - it is fixed
19:19:22 <marja> tarazed: yeah, it's fine
19:19:39 <wilcal> I'm pretty sure I tested the latest CI's in Vbox
19:19:46 <tjandrews> Sorry, minor emergency. jibz: I'm Tom Andrews, AKA "TJ." I'm a farmer in Central New York, USA.
19:20:01 <tarazed> wilcal: you did.
19:20:07 <lewyssmith> I must do the 64-bit Classic DVD tpmorrow; should have time.
19:20:23 <marja> wilcal: I saw you did for the 32bit Live, but you didn't add your results for the 32bit CI
19:20:30 <wilcal> I put two messages on the QA mailiing list
19:20:56 <wilcal> After this meeting I'll fire up my computer again and find out where I'm at
19:21:06 <lewyssmith> reminder: https://pad.riseup.net/p/MageiaPretest
19:21:08 <marja> wilcal: ah, I only looked at the pad.... there are too many mails to read :-(((
19:21:14 <[mbot> [ Riseup Pad ]
19:21:17 <DavidWHodgins> Let's plan on releasing the the pretesting iso images for wider qa testing tomorrow, unless the efi/usb problems can be confirmed as still present
19:21:37 <wilcal> I'll look at it now
19:21:44 <DavidWHodgins> Thanks
19:22:07 <DavidWHodgins> Any other questions or comments about the iso pretesting?
19:22:46 <marja> since when works booting from USB so fast?
19:22:48 <wilcal> not from me
19:22:54 <tarazed> wilcal: Looks like Herman did it - MBR perhaps.
19:23:05 <marja> (to get the boot menu I didn't have time to blink my eyes)
19:23:19 <DavidWHodgins> :-)
19:23:23 <DavidWHodgins> #topic * Testing updates
19:23:23 <lewyssmith> Lucky you!
19:23:35 <marja> lewyssmith: :-)
19:23:48 <lewyssmith> Updates: This is queered by madb being unavailable.
19:24:04 <wilcal> Ya server down
19:24:06 <DavidWHodgins> The good news is that for the first time in a long time, before bugzilla was stopped, there were no updates awaiting testing, that were not awaiting feedback from packagers
19:24:14 <jibz> sorry for the aside ( tjandrews: :D Really ? I can't imagine a field in the center of a city for a farmer... But perhaps I didn't get something ;) I'm not anymore a student, but I'm nothing else for now. (jobless :p) But I will continue my studies.)
19:24:24 <lewyssmith> Bugzilla still works.
19:24:42 <DavidWHodgins> lewyssmith: It's down
19:24:45 <tarazed> Dave made a heroic attack on them and cleared the board.  Temporary lull I guess until everything is back on line.
19:25:09 <wilcal> Ya I've done a successful Vbox Client for both M6 i586 & x86_64 looks good
19:25:25 <lewyssmith> savidw: If so, it worlked after madb went down.
19:25:34 <tjandrews> jibz: Central New York STATE. I'm about a six-hour drive from New York CITY.
19:25:57 <wilcal> BTW all my testing is being done on an M6 x86_64 Nvidia host
19:26:04 <DavidWHodgins> https://bugs.mageia.org is an alias for alamut.mageia.org which was shut down to allow it to be upgraded to Mageia 5
19:26:09 <wilcal> M6 on M6 :-)
19:27:03 <DavidWHodgins> So until the two servers in question are upgraded, or the services moved to systems that are running Mageia 5, there will not be any updates  assigned to qa
19:27:24 <lewyssmith> What were they running before?
19:27:26 <wilcal> pending updates page is not responding
19:27:48 <DavidWHodgins> champange was running Mageia 4. alamut was still running Mageia 1
19:27:54 <lewyssmith> !
19:28:08 <DavidWHodgins> Yeah. Not an acceptable situation
19:28:40 <lewyssmith> Who provoked the scare?
19:28:46 <DavidWHodgins> My message to the council asking for them to be brought offline was public
19:28:58 <wilcal> Ya I saw that.
19:29:02 <DavidWHodgins> I should have used the group address, rather than the mailing list
19:29:09 <wilcal> Sometimes you gotta kick the can to get things going
19:29:19 <DavidWHodgins> True
19:29:24 <lewyssmith> Definitely.
19:29:47 <DavidWHodgins> I certainly got things going. Within a few hours of that message, alamut was hacked
19:29:58 <wilcal> OMG
19:29:59 <DavidWHodgins> That's when it did get shutdown
19:30:21 <DavidWHodgins> Or at least most of the services running on it
19:30:42 <DavidWHodgins> Later, puppet restarted the services, and they had to be shutdown again
19:31:14 <Akien> Mainly because the Council message was reposted on the linuxfr website, which has a big community: http://linuxfr.org/users/coquelicot-bleu/journaux/pas-de-mises-a-jour-de-securite-depuis-5-ans-sur-l-infrastructure-mageia-est-ce-bien-raisonnable
19:31:16 <[mbot> [ Pas de mises à jour de sécurité depuis 5 ans sur linfrastructure Mageia. Est-ce bien raisonnable ? - LinuxFr.org ]
19:31:23 <Akien> 200 comments in two days, lots of trolling in there :)
19:31:46 <tjandrews> Had you not been so public, the response might have been "Yeah, we'll have to get right on that." Then, nothing.
19:31:48 <DavidWHodgins> I'm only sorry that the situation happened at all, not that I made it public.
19:31:51 <Akien> Lots of stress for the French speakers who tried to discuss with those guys, but it's true that it did get things moving.. :)
19:32:36 <DavidWHodgins> Given that it's been almost 5 years since Mageia 1 was end of life, without being public, I have no idea how long it would have taken.
19:32:52 <DavidWHodgins> We have no idea if it was hacked before I went public
19:32:57 <lewyssmith> Why not just within council?
19:33:33 <DavidWHodgins> As identity.mageia.org is on alamut, once things are back up, I'll be pushing for trying to get everyone to change their passwords
19:34:03 <lewyssmith> Their mageia-ID ones?
19:34:39 <marja> lewyssmith: yeah
19:34:49 <lewyssmith> Thanks.
19:34:52 <DavidWHodgins> lewyssmith: I could or should have sent it to the group@council address, rather than to the council ml, but didn't. Didn't think of it at the time. I was pretty emotional when I notice Mageia 1 given all of the remote access bugs that have been fixed since then
19:35:15 <krisNL> passwords were not encrypted on the server?
19:35:48 <marja> krisNL: I understood it's only an issue if you changed your password when a hacker was luring
19:36:12 <DavidWHodgins> They are, but anyone logging in to identity sends the password encrypted by https, it's then passed in clear text to ldap where it's encrypted for comparison to what's in the database
19:36:25 <krisNL> thx
19:36:30 <marja> ouch
19:36:38 <DavidWHodgins> If an intercept was done in between apache and ldap, it would have the password in clear text
19:36:47 <wilcal> So like my password to bugzilla should be changed
19:36:48 <marja> :-(
19:36:53 <wilcal> how about the forums
19:37:02 <marja> wilcal: that's the identity password
19:37:14 <marja> wilcal: forums, wiki, mailing lists
19:37:20 <marja> wilcal: they all use identity
19:37:31 <marja> wilcal: only the blog doesn't
19:37:36 <wilcal> all should be changed right
19:37:45 <DavidWHodgins> Given that you have to login to identity for all Mageia services like bugzilla, mailing lists, etc., everytime your ip address changes, we have to assume the passwords have been captured
19:38:02 <marja> wilcal: you change in identity.mageia.org, that's enough
19:38:03 <lewyssmith> wilcal: But only 1 p/w.
19:38:13 <wilcal> k
19:38:24 <DavidWHodgins> We can not prove they were not, and while we can't prove they were captured, we should assume they were.
19:38:35 <krisNL> we are sure the servers were already cracked?
19:38:47 <DavidWHodgins> No we are not sure
19:39:10 <marja> krisNL: the only thing we're sure of, is that the mailing list archives were deleted
19:39:24 <DavidWHodgins> After I made it public
19:39:25 <lewyssmith> Cracked = fissured, certainly; cracked = hacked - unkown.
19:39:30 <krisNL> geez
19:39:49 <lewyssmith> Benmc: Hello Ben.
19:39:55 <marja> blino tried to undelete them (not sure how far he got)
19:39:58 <Benmc> hiya
19:40:05 <DavidWHodgins> HiYa Benmc
19:40:37 <DavidWHodgins> For those wondering how this happened, I'll explain
19:40:47 <lewyssmith> Please do!
19:40:57 <DavidWHodgins> First, http://madb.mageia.org/tools/updates was not responding for me
19:41:04 <DavidWHodgins> It was for sysadmins
19:41:21 <tjandrews> So is there any talk of doing this sort of server update more regularly from now on?
19:41:51 <DavidWHodgins> That was traced to ns0.mageia.org having the correct address for the site, but ns1.mageia.org still having an old one, so whether it worked or not depended on which name server was used
19:42:52 <DavidWHodgins> That was traced to puppet not running on krampouzh (sp?), where ns1.mageia.org runs
19:43:20 <DavidWHodgins> So then I started looking at xymon.mageia.org to see if any of the other servers still had old files
19:44:04 <DavidWHodgins> I was shocked when I saw champhange was still running Mageia 4, which has been end of life for about 18 months
19:44:13 <DavidWHodgins> Then I saw alamut
19:44:18 <DavidWHodgins> Mageia 1
19:44:29 <DavidWHodgins> EOL for almost 5 years!
19:44:34 <DavidWHodgins> I felt sick
19:44:53 <DavidWHodgins> Especially when I looked at what services were running on it
19:45:22 <lewyssmith> BTW What is puppet?
19:45:59 <DavidWHodgins> puppet is the service used to install packages and changes to config files on all of the servers, keeping things in sync etc.
19:46:01 <wilcal> https://en.wikipedia.org/wiki/Puppet_(software)
19:46:15 <Benmc> (afk for a long while....)
19:46:56 <DavidWHodgins> I paused, and rewrote my message to the council several times, to try and take some of the emotion out of it
19:47:02 <marja> Benmc: bye :-)
19:47:16 <lewyssmith> Goodbye Marja; nice to see you.
19:47:18 <DavidWHodgins> Finally decided to send the message that I did send
19:47:38 <krisNL> may I ask why the servers were unmaintained for so long?
19:47:38 <marja> lewyssmith: :-)
19:47:48 <wilcal> Puppet's another one of those Career applications
19:47:51 <DavidWHodgins> Then as they say, the Sh** hit the fan
19:48:00 <lewyssmith> |I clearly misunderstood; the sentiment stands].
19:48:14 <marja> krisNL: sysadmins being very short on time, issues with hardware...
19:48:19 <wilcal> BTW this is what QA is for
19:48:45 <wilcal> If you've got good QA you find the "Sh** hit"
19:49:04 <krisNL> marja: thats an explanation, but not an excuse.
19:49:08 <DavidWHodgins> krisNL: Doesn't matter now. It's done. We just have to get the situation fixed, and make sure nothing like it happens again.
19:49:25 <lewyssmith> Learning the hard way.
19:49:36 <krisNL> so we need more sysadmins with more time
19:49:36 <DavidWHodgins> The past is the past. We have to focus on the future, not worry about placing blame etc.
19:49:45 <DavidWHodgins> Pretty much
19:50:01 <lewyssmith> -2 Agree.
19:50:18 <marja> krisNL: and, probably too: that originally the infrastructure was designed for the sysadmins to have access to everything.... so someone who'd be a great sysadmin for the server with the wiki, or Bugzilla, couldn't get sysadmin access there, because it was all or nothing
19:51:01 <marja> krisNL: tmb started working on changing that, so that sysadmin apprentices can gradually get more access
19:51:02 <DavidWHodgins> As I understand it, that's still the case. Anyone who is a full sysadmin has full root access to all servers
19:51:35 <marja> DavidWHodgins: yeah, but whatshisname, the colleague of pterjan, he has less access
19:51:39 <DavidWHodgins> Hence, it's one of the hardest teams to join, as trust has to be built first
19:51:52 <marja> DavidWHodgins: so it is changing
19:52:07 <DavidWHodgins> Yes, it is changing.
19:52:44 <DavidWHodgins> But very difficult to implement, especially when such old software was involved
19:53:19 <DavidWHodgins> So that's the situation. Any questions or comments on that?
19:53:25 <wilcal> Not from me
19:53:46 <wilcal> we can still move pretest -> test while this is all going on right?
19:54:23 <DavidWHodgins> Yes. Those directories are on rabbit, so not affected
19:54:28 <marja> wilcal: yes.... the only problem is, that bugs can't be filed
19:54:48 <DavidWHodgins> They can still be reported on the ml and pad
19:54:53 <wilcal> Back up by Mondayish?
19:54:56 <marja> DavidWHodgins: indeed
19:55:07 <DavidWHodgins> wilcal: No idea
19:55:42 <krisNL> and rabbit is secured?
19:55:56 <marja> wilcal: iinm, sysadmins will first try to get mirrorlist back up, but bugzilla will be back before the wiki
19:55:57 <DavidWHodgins> I don't know, and doubt the sysadmins really know right now. Better for them to actually work on it than worry about answering questions about eta, etc. right now.
19:56:17 <DavidWHodgins> krisNL: As far as we know.
19:56:37 <marja> and they are working on upgrading both Bugzilla and the wiki
19:56:59 <DavidWHodgins> As far as we know, this has had no impact on rpm packages or iso images, but we have no way to be 100% positive
19:57:36 <wilcal> I have the M5/Cauldron repo here
19:57:38 <DavidWHodgins> There is enough review of changes that it's extremely unlikely, but not impossible
19:58:14 <DavidWHodgins> Package changes have to go through the build system, and that is watched pretty closely
19:58:34 <DavidWHodgins> iso building is done manually, not on either of the affected systems
19:58:47 <wilcal> 160,000 files & 361GB :-0
19:59:03 <DavidWHodgins> wilcal: Take it you don't have the debug repos
19:59:44 <DavidWHodgins> Anyway, let's move on
19:59:48 <DavidWHodgins> #topic * Anything else?
20:00:03 <DavidWHodgins> So anyone have any other topics to discuss?
20:00:03 <wilcal> everything in 5/6 media
20:00:22 <DavidWHodgins> debug too?
20:00:26 <wilcal> Yes
20:00:31 <wilcal> looks like it
20:00:49 <wilcal> Think we'll need it?
20:01:21 <wilcal> I mirror everything in mirrors.kernel.org for M5 and Cauldron
20:01:28 <DavidWHodgins> Been a while since I had my own copy of the repos. Lost that when I lost my old machine due to a power supply failure which took the mg and 2 of it's 3 hard drives with it
20:01:42 <DavidWHodgins> s /mg/mb
20:01:56 <wilcal> I actually have two copies one on this server and another here that backs that up
20:02:20 <DavidWHodgins> I had most stuff backed up, but not the repos
20:02:35 <DavidWHodgins> Anyway, looks like countdown time
20:02:38 <DavidWHodgins> t - 5
20:02:39 <wilcal> 361GB is a big download
20:02:43 <wilcal> bye all
20:02:44 <DavidWHodgins> 4
20:02:50 <DavidWHodgins> 3
20:02:50 <lewyssmith> Goodbye everyone.
20:02:53 <DavidWHodgins> 2
20:02:57 <DavidWHodgins> 1
20:03:02 <DavidWHodgins> Thanks for coming everyone
20:03:05 <tarazed> Solong.
20:03:11 <DavidWHodgins> #endmeeting