Linux Mint: Hacked 2.0 (Official Unofficial Notice) UPDATED: 2016-07-18

UPDATE: 20160718T12Z

Torrent server has been reset and  a  ‘private dht seed’ made greatly increasing the security of the seed pool and  denying most WAREZ trackers from leeching or  connecting back.

paste.linuxmint.com also seems to be fully back up and functioning fully / correctly (irc logs in #linux-dev this morning show both were reset and restarted)

GPG keys are again retrievable normally  and the proper commands for retrieving them and checking the sha256sum files updated to reflect proper / best practice methods.

Changes to overall mentality are still likely needed but a start in the proper direction seems to finally have been made.  Updates to follow as they present themselves.

It is with deep despair that today I have to write that at least portions of the Linux Mint Infrastructure seem to have  come under a coordinated attack and that the Development Team has to date NOT responded to subtle non standard channel requests for info on this.

Late last night, US East Coast time, it became apparent to several of the users presently in #linuxmint-help on the Linux Mint IRC Network housed over at  ircs://irc.spotchat.org  that links returning from  https://paste.linuxmint.com were acting ‘funny’ or  not even loading, later it became that the main page irregardless of the /view/foo   was no longer loading without the aid of a VPN with an Endpoint  from within the EU / Asian Geographic area of the globe.

Some snippets of these that became alarming follow:

/usr/local/bin/pastebin   is supposed to be  .noarch aka does not care if the system is 32 or 64 bit  as it runs same regardless and only needs added / used once in the build process, however if you run a ls -la OR sha256sum operation on the file you will see one of 3 outcomes

 ls -la /usr/local/bin/pastebin     No such file or directory

 

sha256sum /usr/local/bin/pastebin

5e11507cacfa516b3c2e0610cf3d437b07aeaddb388bcf92a89f19b1bca54d55

 

sha256sum /usr/local/bin/pastebin

74901a0a6884104ccaa6fba5858622bcd7603bf3b666ae5db2fddd9a38b2ca16 /usr/local/bin/pastebin (valid hash)

 

Furthermore: The troubled https://linuxmint.com/verify.php fails to indicate how to download the sha256sum.txt and sha256sum.txt.gpg files are they need a non standard way  i.e. wget or cURL.

Complicating things is the fact that since around 930-945 PM US EST last night (7-16-16) the following commands were failing:

gpg –list-keys –with-fingerprint

gpg –verify sha256sum.txt.gpg sha256sum.txt

gpg –recv-keys A25BAE09

gpg –recv-keys A25BAE09

In large part because Linux Mint comes WITHOUT a default –keyserver, which I can understand the desire to not force any one or series of keyservers on the user, but knowing this and that Clem the fearless leader only  directly pushes to keyserver.ubuntu.com  those above –recv-keys commands should include –keyserver keyserver.ubuntu.com .

But all that aside the keys are not even obtainable from:    keyserver.ubuntu.com, possibly in part to Ubuntu’s response to their own hack of their Forums: Ubuntu’s Forums Hacked, 2 Million usernames stolen .

Due to this:  last week I emailed Clem about ways to clean up the readability and even made the following gist paste available for use or  straight scraping.

Linux Mint 18 ISO GPG validation steps

This went on seemingly deaf ears after an apparent initial response of sure we can do that.

UPDATE:  AS OF:  17 Jul 2016 1845UTC   whois record for http://paste.linuxmint.com  indicates it is UP FOR SALE.  Discontinue use immediately.

AS OF 2000utc  17 Jul 2016 , 18 hours after community confirmation of compromise no word what so ever from the dev team.

Some other enlightening links for the scope of shit going down unanswered:

http://status.linuxmint.com

http://65.19.183.167/ameridea  look at Screenshots on that page

https://www.irccloud.com/pastebin/2Rfo6Oh9

https://www.irccloud.com/pastebin/7TG8pwGW (contains valid hash for the script that is /usr/local/bin/pastebin )

http://www.omgubuntu.co.uk/2016/07/ubuntu-forums-hacked-2-million-usernames-stolen  (mentioned above as well)

whois |  traceroute | mtr | dig      141.8.244.93 and compare to same for 208.92.233.240

https://gist.github.com/linux-modder/053f1da7bf247aa448a128ad7799557d  (decrypted copy of email originally sent to Clem  root@linuxmint.com at 1300UTC 3 Jul 2016.)

http://network-tools.com/default.asp?prog=dnsrec&host=paste.linuxmint.comand

 

And some IRC chat logs that are telling: Times unless stated otherwise in logs are  US EST (utc -4)

FROM ircs://irc.spotchat.org/#linuxmint-dev

[Sat, 16 July ://: 19:18:03] .:linuxmodder:. ANYONE from admin@ or root@ here? seems a major set of issues with paste.lm.c
[Sat, 16 July ://: 19:19:48] «— notis (notis@SpotChat-bdrdnk.dyn.forthnet.gr) has Quit (Quit: Leaving)
[Sat, 16 July ://: 19:29:39] .:r00t:. Said that in here ~6 hours ago **
[Sat, 16 July ://: 19:31:53] .:linuxmodder:. r00t, a paste sourced from -help is seemingly proxy aware and asking for openkeychain for some and for my setupvpn browser login creds SOMEONE from the infra side needs to do some serious checking
[Sat, 16 July ://: 19:33:44] .:r00t:. It seems normal to me now. But then, I don’t use a proxy
[Sat, 16 July ://: 19:34:34] .:linuxmodder:. what seems normal now?
[Sat, 16 July ://: 19:34:59] .:r00t:. I mean I click on a link and it shows the text
[Sat, 16 July ://: 19:35:06] .:r00t:. Normal
[Sat, 16 July ://: 19:35:29] .:r00t:. I talked about something being strange a few hours ago, and they must’ve fixed it
[Sat, 16 July ://: 19:36:14] .:linuxmodder:. nope does not look that way
[Sat, 16 July ://: 19:37:01] .:r00t:. I know what I’m seeing right in front of me. Maybe your proxy is doing something
[Sat, 16 July ://: 19:40:29] .:r00t:. Besides, only clem has access to the servers so nobody here can do anything about it (if it is actually messed up)
[Sat, 16 July ://: 19:41:37] .:linuxmodder:. r00t, that’s just it I’m not over proxy (besides tor on the nic)
[Sat, 16 July ://: 19:48:42] —» calexil_SteamBox (jd@SpotChat-e6ogl5.de.comcast.net) has Joined #linuxmint-dev
[Sat, 16 July ://: 19:52:29] —» LunarEclipse120 (LunarEclips@unaffiliated/lunareclipse120) has Joined #linuxmint-dev
[Sat, 16 July ://: 19:56:08] .:linuxmodder:. calexil, you have a way to wake / get hold of clem ?
[Sat, 16 July ://: 19:56:35] .:linuxmodder:. i know its like 2am there but think its warranted

** [Sat, 16 July ://: 13:55:44] .:r00t:. For some reason paste.linuxmint.com makes someone download the paste, and makes it a .bin extension

---

[Sat, 16 July ://: 21:27:12] .:linuxmodder:. calexil, ??

[Sat, 16 July ://: 21:28:21] —» Testing567 (Testing567@SpotChat-hegnee.dyn.optonline.net) has Joined #linuxmint-dev
[Sat, 16 July ://: 21:30:48] «— bario (bario@I.am.not.your.friend) has Quit (Quit: Leaving)
[Sat, 16 July ://: 21:30:55] <JosephM> linuxmodder: he can’t do anything. Clem will be here when he gets here
[Sat, 16 July ://: 21:31:02] »» maccarone is now known as bario
[Sat, 16 July ://: 21:31:28] .:linuxmodder:. RECHECK servers and gpg keys Something is —-seriously not right http://65.19.187.163/ameridea/ 2nd-4th screenshots

---

[Sat, 16 July ://: 23:56:27] .:linuxmodder:. when clem gets up Purge the servers

[Sat, 16 July ://: 23:56:44] .:linuxmodder:. routing from traceroute is using a known hacker spoof route

---

[Sun, 17 July ://: 03:42:16] .:linuxmodder:. your infra and keys are going to shit you gonna do anything yet?

---

[Sun, 17 July ://: 15:07:05] 19.:2823linuxmodder19:. someone needs to come out and explain the loss of paste..linuxmint.com and other oddities today

---

---

 

FROM ircs://irc.spotchat.org/#linuxmint-help (again all times shown or utc -4 unless specified)

[Sun, 17 July ://: 16:22:38] 19.:2823linuxmodder19:. mtn, so we are gonna forget or ignore that facts that BOTH paste.linuxmint.com and linuxmint.com BOTH show NX on a whois lookup?
[Sun, 17 July ://: 16:22:41] 31-31-19-19» 19juanjo (juanjo@SpotChat-nfu.6c1.71.90.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 16:22:48] 28.:1924mtn28:. hippo: what, you just want to start the update manager from the command line?
[Sun, 17 July ://: 16:22:49] 31-31-19-19» 19c (c@SpotChat-6f2ide.pa.comcast.net19) has Joined #linuxmint-help
[Sun, 17 July ://: 16:22:54] 26«26-31-31- 26juanjo (juanjo@SpotChat-nfu.6c1.71.90.IP26) has Quit (Quit: Leaving26)
[Sun, 17 July ://: 16:22:56] 26«26-31-31- 26c (c@SpotChat-6f2ide.pa.comcast.net26) has Quit (Connection closed26)
[Sun, 17 July ://: 16:22:59] 31-31-19-19» 19bona (bona@SpotChat-n7q6ik.p9ot.juqq.120b.2a02.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 16:23:07] 04<mtn04>04 linuxmodder: I have to say it does not matter to me

---

[Sun, 17 July ://: 15:48:32] 19.:2823linuxmodder19:. mtn, where the hell is clem with a response to last night and today’s oddities?/
[Sun, 17 July ://: 15:48:49] 19.:2823linuxmodder19:. or anyone form the dev team for that matter

---

[Sun, 17 July ://: 15:51:01] 04<mtn04>04 linuxmodder: which oddities?
[Sun, 17 July ://: 15:51:03] 31-31-19-19» 19shadowmaster (shadowmaste@SpotChat-074v1o.res.rr.com19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:51:25] 31-31-19-19» 19Guest8229 (john@SpotChat-5iv951.east.verizon.net19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:51:31] 19.:2823linuxmodder19:. mtn, in a few minutes you can read about them from my blog
[Sun, 17 July ://: 15:51:44] 26«26-31-31- 26SimonNL (Simon@i.am.the.true.idiot26) has Quit (Quit: Leaving (Close)__If I have said something clever. my apologies\o26)
[Sun, 17 July ://: 15:51:48] 31-31-19-19» 19oem (oem@SpotChat-rvs.utr.20.159.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:51:50] 31-31-19-19» 19notis (notis@SpotChat-r0bbk5.dyn.forthnet.gr19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:52:33] 19.:2823linuxmodder19:. but some quick hits http://paste.linuxmint.com has been beeing blocked from all BUT eastern EU / Asian IPs (or vpns) and is now according to whois up for sale
[Sun, 17 July ://: 15:52:34] 31-31-19-19» 19mint (mint@SpotChat-mei.265.172.178.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:52:38] 26«26-31-31- 26mint (mint@SpotChat-mei.265.172.178.IP26) has Quit (Connection closed26)
[Sun, 17 July ://: 15:52:47] 19.:2823linuxmodder19:. it has been hanging and timing out for over 16 hours

---

[Sun, 17 July ://: 15:51:31] 19.:2823linuxmodder19:. mtn, in a few minutes you can read about them from my blog
[Sun, 17 July ://: 15:51:44] 26«26-31-31- 26SimonNL (Simon@i.am.the.true.idiot26) has Quit (Quit: Leaving (Close)__If I have said something clever. my apologies\o26)
[Sun, 17 July ://: 15:51:48] 31-31-19-19» 19oem (oem@SpotChat-rvs.utr.20.159.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:51:50] 31-31-19-19» 19notis (notis@SpotChat-r0bbk5.dyn.forthnet.gr19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:52:33] 19.:2823linuxmodder19:. but some quick hits http://paste.linuxmint.com has been beeing blocked from all BUT eastern EU / Asian IPs (or vpns) and is now according to whois up for sale
[Sun, 17 July ://: 15:52:34] 31-31-19-19» 19mint (mint@SpotChat-mei.265.172.178.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:52:38] 26«26-31-31- 26mint (mint@SpotChat-mei.265.172.178.IP26) has Quit (Connection closed26)
[Sun, 17 July ://: 15:52:47] 19.:2823linuxmodder19:. it has been hanging and timing out for over 16 hours
[Sun, 17 July ://: 15:52:48] 26«26-31-31- 26oem (oem@SpotChat-rvs.utr.20.159.IP26) has Quit (Quit: Sto andando via26)
[Sun, 17 July ://: 15:52:49] 26«26-31-31- 26shadowmaster (shadowmaste@SpotChat-074v1o.res.rr.com26) has Quit (Quit: Leaving26)
[Sun, 17 July ://: 15:52:55] 31-31-19-19» 19zodian (zodian@SpotChat-pjirop.home.otenet.gr19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:52:57] 31-31-19-19» 19neosunaru (neosunaru@SpotChat-operb0.u42h.g04k.c1a0.2a00.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:53:04] 26«26-31-31- 26neosunaru (neosunaru@SpotChat-operb0.u42h.g04k.c1a0.2a00.IP26) has Quit (Quit: Leaving26)
[Sun, 17 July ://: 15:53:06] 26«26-31-31- 26martin_ (martin@SpotChat-59s6tu.starnet.cz26) has Quit (Ping timeout: 121 seconds26)
[Sun, 17 July ://: 15:53:15] 31-31-19-19» 19zodian_ (zodian@SpotChat-pjirop.home.otenet.gr19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:53:18] 19.:2823linuxmodder19:. gpg keys for BOTH 17.x and 18 have been unreliably recieveable for same period
[Sun, 17 July ://: 15:53:21] 19.:2823linuxmodder19:. and others
[Sun, 17 July ://: 15:53:26] 26«26-31-31- 26zodian_ (zodian@SpotChat-pjirop.home.otenet.gr26) has Quit (Quit: Leaving26)
[Sun, 17 July ://: 15:53:31] 31-31-19-19» 19ciber (ciber@SpotChat-sue.jq7.236.201.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 15:53:34] 26«26-31-31- 26zodian (zodian@SpotChat-pjirop.home.otenet.gr26) has Quit (Quit: Leaving26)
[Sun, 17 July ://: 15:53:38] 19.:2823linuxmodder19:. cat is out of the bag FOLKS

---

[Sun, 17 July ://: 15:55:22] 19.:2823linuxmodder19:. I personally can not recommend or vouch for the sanity / integrity of servers / mirrors

[Sun, 17 July ://: 15:56:45] 04<mtn04>04 linuxmodder: not good at all 😦

[Sun, 17 July ://: 15:57:20] 19.:2823linuxmodder19:. mtn, no shit ( pardon the language) but I’ve been helping others pinpoint that for over 18 hours on a distro I don’t even daily use
[Sun, 17 July ://: 15:57:32] 19.:2823linuxmodder19:. and been crickets from clem et al
[Sun, 17 July ://: 15:58:16] 28.:1924bario28:. where can I read about these oddities?
[Sun, 17 July ://: 15:58:22] 19.:2823linuxmodder19:. I’ve been alerting folks around the globe and across projects more than I do for me beloved Fedora Release day
[Sun, 17 July ://: 15:58:28] 28.:1924javier_28:. Im Javier, nice to meet you. Im using Mint 18 a 2 days ago and Im really impresed, almost love it
[Sun, 17 July ://: 15:58:32] 26«26-31-31- 26Dragoon (cyber_drago@SpotChat-os6.ve6.7.179.IP26) has Quit (Quit: Leaving26)
[Sun, 17 July ://: 15:58:36] 19.:2823linuxmodder19:. bario, try loading http://paste.linuxmint.com
[Sun, 17 July ://: 15:58:43] 26«26-31-31- 26ciber (ciber@SpotChat-sue.jq7.236.201.IP26) has Quit (Quit: Leaving26)
[Sun, 17 July ://: 15:58:45] 19.:2823linuxmodder19:. whois paste.linuxmint.com in a terminal

[Sun, 17 July ://: 16:02:45] 19.:2823linuxmodder19:. traceroute 208.92.233.240 and see the blocks and redirects

---

[Sun, 17 July ://: 16:45:30] 04<revdjenk04>04 linuxmodder: I am not seeing any signs, notifications, alerts about this at all.
[Sun, 17 July ://: 16:45:51] 28.:1924revdjenk28:. paste is down, only
[Sun, 17 July ://: 16:45:52] 19.:2823linuxmodder19:. that is the problem
[Sun, 17 July ://: 16:45:57] 31-31-19-19» 19iceunicorn (iceunicorn@SpotChat-o95.a02.203.67.IP19) has Joined #linuxmint-help
[Sun, 17 July ://: 16:46:05] 26«26-31-31- 26aron (aron@SpotChat-9t323c.078h.76to.a601.2605.IP26) has Quit (Quit: Leaving26)
[Sun, 17 July ://: 16:46:13] 19.:2823linuxmodder19:. revdjenk, my blog post with the signs for those unaware will be up shortly.