user tip Looks like my jr domain Mailgun acct was hacked

Went to my fotmd.com jamroom site Mailgun log today and saw this (see screenshot).
Looks like last night someone sent out almost 2 million spam emails from my Mailgun domain account associated with my jamroom network. GaaaCK! =8-0
I just opened a Mailgun ticket, hope I don't get my site mail shut down over this. And I hope they can help me! I have no idea who this was or how it happened.
All I know is that the day prior to this, I received a response from Mailgun about a ticket that I supposedly opened with them requesting a dedicated IP be added to my account. I did NOT request that, yet there was a real actual ticket open in MG requesting it. In the ticket, MG had responded to the person and asked for more info about how many and what kind of mail they were intending to send on the IP. I replied yesterday to the MG ticket saying that *I* did not request this.

Apparently MG must have gone ahead and granted it or something, and now the next day almost 2 million spam messages were sent out from my domain. Yikes.
Anyway, I just now opened a new ticket with MG telling them that was NOT me and asking for their help in un-hacking my account. I hope my MG account doesn't get blocked or dumped because of this!
I'm posting this here just in case another JR member is experiencing this or as a warning if they get a notice from MG about a ticket they supposedly opened asking for a dedicated IP... beware if that happens, contact MG immediately!
I just now changed my MG password, but am waiting to hear back from Mailgun with my new ticket.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 10/01/18 06:38:51PM

Sorry to hear that Strumelia.

Please keep us updated on what you've been recommended to do. I just went there now to change my own password and it seams like that feature is locked at the moment.

Thanks Michael. I've been spending a while now attending to a bunch of stuff the Mailgun guy recommended in his ticket response.
I've now done the following:
1) changed my Mailgun account password.
2) reset my Mailgun Private API key (and then filled the new one in on my JR server settings and then ran a test email which worked)
3) I enabled 2factor authentication in my mailgun account, using my current password program authenticator App for my android phone. (a total pain in the butt but I suppose now a necessary evil).
4) the mailgun guy also told me two IP addresses from which the spam was being sent and I banned the ranges of both.

================

The last suggestion from the mailgun support guy:
This is the one thing I may have screwed up: resetting my Mailgun domain "SMTP credentials" passwords. I wasn't quite sure what they WERE so I reset them with my main JR password. Now it seems I can no longer send or receive emails through my main ___@fotmd.com address, my JR email support "Test Email address" sending FROM my main @fotmd email is now no longer working since I did this it seems. (was still working still after I reset the API though)
Here's a screenshot from the Mailgun Help blog showing the SMTP settings area I'm talking about, and mine has a setting both for my main @fotmd.com email and also "postmaster@___". I'm not sure whether those PWs need to be the same as my Mailgun Account main PW, and/or do they need to be the same as _WHICH_ PWs on my Jamroom end? Do I find the PWs for those items on my Jamroom settings and need to update them?
Right now I'm no longer getting emails from OR to my fotmd.com email address aliases.. My JR mail log seems to confirm this stoppage as well... due to my messing 'something' up I think with the smtp MG passwords, or else by resetting those and not synching the new pws to something on the other end in JR. ?


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 06/23/18 07:48:44PM

P.S. resetting the passwords for those mailgun "Manage SMTP credentials" domain areas (as in the screenshot) was the only thing I attempted to do but wasn't sure about what my pws HAD been (wouldn't let me see the prior pws) and how to coordinate them on the Jamroom side. Things seemed to be working ok until I messed with them. My JR System Check is all green. At this point my site mail/notifications are no longer being sent or received I believe. Everything seemed to be working until I tried to fuss with the smtp credentials in Mailgun.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 06/23/18 07:46:50PM

yeah jamroom side doesnt need your password, just your API key, so if you've changed that and updated you should be good to go.

Let us know if you see anything weird though.

On the bright site, now you know you can send two million emails in a day if you find yourself wanting to

Um, well I told the Mailgun fellow I was not going to be paying for those 2 million emails... =8-0 Gosh I hope they don't bill me!

OK, so I simply reset new PWs for those MG smtp credential settings.
Is there a way to refresh my JR email delivery log? I'm not seeing anything newer than like 7 hours ago on it. This despite my having done several actions on my site that should have triggered notifications. I also restarted my JR server and Apache, just in case.

And now I have to go to bed. :-\


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 06/23/18 08:50:29PM

Hmm, my site notifications are still not working.
Is there a way to refresh my JR email log? it shows no mails sent since yesterday late afternoon. Or else it's just that nothing has been sent out since then, despite all kinds of site activity that should have triggered notifications and emails.

I should note that private messages are still functioning within the site, though not the notifications for them.
BTW...My other JR site domain on the same server, pennywhistleclub.com has all its notifications still working fine (that domain was not the one attacked by the hacker) -I just tested it.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015

OK, suddenly 10 minutes after this last post something kicked in and I see a few notifications going through.

Somebody did 'something' to fix it, either from the JR end or from the Mailgun guys whom I also wrote to again 20 minutes ago. Maybe my domain had been deactivated/blocked by MG?
So hold on.... it seems to be working again all of a sudden. Miracle!... keeping fingers crossed and will report back here again in an hour or so.

BTW the MG fellow said he applied a credit to my account for the cost of the spam messages. (whew, or else I'd be maybe posting to here from jail in the future)


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 06/24/18 09:43:47AM

Wow!! Thank you for sharing this with us. I will be making a thorough check of our MG account. Sorry, you had to deal with all this... taking one for the team is not always fun. We sure do appreciate your detailed reports. Thanks again!!


--
The Patria Company - patriaco.com / quality-trades.com / a-t.life - doing Jamroom since v3

Thanks Patria,
BTW the mailgun support guy said that THE most important security measure we can take is to activate the "2FA" (2 factor authentication) for our Mailgun account. This I did on my Android phone by downloading an authenticator app.

-It looks like my normal site notifications are now pouring in from last night and this morning- the faucet has been turned back on by ...someone... Yay! :D


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 06/24/18 10:22:49AM

So- in retrospect, several RED FLAGS to look out for as a hacker is working their way into your mailgun account:

1- getting emails from a Mailgun support person saying your domain needs to be validated... even though your domains are already validated. I ignored these emails thinking they were from a spammer.. but they were perhaps indicative of the hacker being in the process of infiltrating my account.
2- getting a support email from Mailgun responding to a support ticket request that you supposedly made, asking for a new Dedicated IP on your account. Big red flag!! This support ticket was actually created in my mailgun account by the hacker, and you must immediately contact Mailgun support and report this as a hacker.
3- getting a friendly note from mailgun suggesting you might want to 'upgrade' your account to higher capacity... this was sent to me about a day after the 2 million spam emails had been sent out by the hacker. =8-o

And it's always wise to glance at your mailgun LOG each day for each domain, so that you can see within 24 hours whether a giant mountain of spam has been sent out. Note that these spam emails did not show up on my Jamroom email log... only in my Mailgun account Logs, on the MG site.

My notifications seem to be turned on again and working normally, so I'm going to mark this thread as a "User Tip". Hopefully it will be of some use to somebody else at some point.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 06/24/18 11:31:54AM

I received this from the Mailgun Support:
"Last night after you mentioned completing the recommended steps, we re-enabled the domains and account for you so that you could resume sending. As such, the site notifications should indeed be working as we're no longer blocking any traffic on the account."
...So Mailgun had indeed disabled my domains and account from sending any emails out (not surprising, but they mentioned nothing about that, nor mentioned when they turned it back on). I guess they simply waited until I told them I had implemented the security suggestions to 'turn me back on'.
Good stuff to be aware of.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015

Thanks Strumelia. Horrible experience im sure.

Thanks Michael. As hacking things go, it could have been something much worse I'm sure. I'm choosing to see it as a good learning experience, since no irreparable damage seems to have been done.

BTW just to be extra safe, I also changed my PW here in my jamroom account and also on my jr sites as Master Admin.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 06/25/18 07:45:38AM

smart. you can generate a new SFTP password for your server also if you like from here:
JAMROOM PROFILE -> DASHBOARD -> HOSTING -> SERVER -> SERVER SETTINGS
https://www.jamroom.net/strumelia/hosting/server_config/11

I think I will go ahead and do that Michael- thanks for the suggestion! :D
Oh, a question on that first- will I need to plug that new SFTP password into any other place- like in my JR modules or in my Mailgun account? (Just need to know if I have to fill the new pw in elsewhere or not, before I go ahead and change it)


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015

no. Thats just for when you want to connect via SFTP to your server. Stuff like adjusting skins via the filesystem, or adding a line of CSS to one of the css files.

I don't think you do that much, preferring to use the TEMPLATES tab of the ACP for your alterations.

Actually, although I prefer making changes through the ACP, I do occasionally have to make changes to css files or add/edit a file directly in my server via SFTP, using Filezilla. So, good!- I will go into my Filezilla settings and look for that PW and change it there as well- this might not have occurred to me until much later on when I could not connect on Filezilla and had perhaps forgotten about having changed the SFTP password. Thanks, will do!


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015

@strumelia

Thanks for all that.

Reporting in from the Peoples' Republic (here for a holiday of sorts) it seems appropriate to suggest that "RED FLAGS" for hacking intrusions might be a useful part of the JR Documentation, if it does not serve as "support" for hackers just as much as it does for site owners. Can we (whoever "we" may be) win the revolution?!


--
PJ Matthews, Kyoto
Migrated from Ning 2.0. Now at Jamroom 6 beta and using Jamroom Hosting for The Research Cooperative (researchcooperative.org)

Sorry to hear this happened to you Strumelia, but YES - 2FA is really needed to protect your account - especially when it is a "high value" target like your email sending account. Glad you got that enabled - I'd recommended everyone set it up on their account.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net