Coeur Business Group IT Audit Skewed; Brower is gone, why isn’t Joe Noel?

UPDATE 11/2015: City IT Department is now, apparently, hiring back the positions that Joe Noel and his very expensive IT Audit insisted they did not need. Again, why does Joe Noel still work for the City of Ocala? These positions are certainly crucial to having a large secured network. Definitely won’t see his perceived savings now… He should be let go.

DATABASE ADMINISTRATOR – Full-Time – $55,800.00 – $95,500.00 Annually
IT – NETWORK SYSTEMS ENGINEER-  Full-Time – $55,800.00 – $95,500.00 Annually
IT NETWORK SECURITY OFFICER – Full-Time – $55,800.00 – $95,500.00 Annually

This was sent to all the council members just before the August 4th meeting when
Joe Noel was slated to give council an update for this past year.

Council Members,

With Joe Noel presenting his information tomorrow night (August 4th, 2015) about the current state of the IT Department and how the last year’s Coeur Group’s changes have impacted the City of Ocala, please keep these facts in mind: Joe Noel intentionally allowed Council to be misled in order to get his agenda pushed through, as well as to cover up his own failed systems that were under his control.

Some of the failed items that Coeur Group revealed where areas that Joe Noel himself was managing, not IT, such as the failed door lock system where City Hall was randomly left unlocked for days each month or so, printing pin numbers in plain sight on security badges, using consecutive PIN numbers, not offering two-factor authentication at the start and much more …

Other areas, such as not using strong passwords, were mandated down from the Horst AND Brower against IT’s VERY strong recommendations.

These are small things in a much bigger picture that you can read about in the links below. Joe Noel and the Coeur Group have not done a service to our City, they have cost the city our tax payer’s money and valuable time in my opinion…

I encourage you City Council to read through this all, I know it’s a lot to digest but it is important. More than 15,000 people looked at my last posts, I truly hope you take the time to understand what really happened last year and make it right.

Unethical uhn-eth-i-kuh l

adjective
1. lacking moral principles; unwilling to adhere to proper rules of conduct.
2. not in accord with the standards of a profession.

Matt Brower is gone, why isn’t Joe Noel?

Hasn’t he more or less provided false information for the IT audit and lied to Council in order to get a much larger salary. If you let what he has done go unaccounted for, you are only enabling more employees to ride roughshod over the City Council in the future and costing the City more taxpayers dollars…

“The assessment’s estimated savings of $6.7M has not yet been realized due to the unanticipated high cost of cloud and managed services.” – JOE NOEL (ASSISTANT IT DIRECTOR)

WHAT?  What truly happened here is that Joe Noel found out that the Infrastructure employees he terminated had already saved the city tons of money moving into our own cloud network, and that moving it would cost more than letting it sit on the hardware that we purchased. That purchase which saved the City of Ocala almost $2M dollars on some OEU projects in a single year that Joe Noel was spearheading at Ocala Electric. (Ironically, we saved the City the cost of the entire IT Budget according to Joe’s numbers, by keeping Joe Noel in check) They wanted to buy physical servers for several of their projects. That’s how our current ESX Stack (Our in-house cloud network) came into existence. We researched technology and ended up buying the ESX system that we could use for, not only their projects, but also for other areas of the City that had needs for servers for their projects, including the active directory infrastructure.  We bought 6 servers that allowed us to run more than a 120+ virtual servers in our own in-house cloud. That $2M in savings alone, plus the other savings by going virtual, should have carried our network engineering expertise through this audit instead of just casting us out as it did.

“Recommendation to reduce staff proved to be detrimental to network health, security and level of service  provided …” – JOE NOEL (ASSISTANT IT DIRECTOR)

So … why is he still there?

Joe says, they still need to complete transition  of  all  our  2003 servers to 2008 … This should have been done no later than July 14th, 2015 … Congratulations, Joe is providing a huge security risk.

Running 2003 servers is a huge security risk. This is another thing that we were actively working on before the Coeur Group and Joe interrupted everything.  According to Microsoft: Windows Server 2003 extended support ended on July 14, 2015. Microsoft will no longer issue security updates for any version of Windows Server 2003. In the case of Windows Server 2003, antimalware support ends when the server becomes an unsupported product. Microsoft is not extending the issuance of antimalware protections for organizations not making the July 14 deadline. If he has not met this deadline, which obviously he hasn’t because it was in his presentation, Joe has provided you a huge security risk on your network AND it’s public knowledge because he presented it to council in a public forum! For that matter, why stop at 2008 Joe, since we are now well into 2015, maybe you should just upgrade everything to Server 2012, if the applications can handle it; the Active Directory Forest and Domain Controllers I built certainly can handle it… then there’s the compliance issue, as there is legislation that is very specific about whether organizations are meeting their responsibilities if they are running unsupported software.

No really … why is he still there?

Let’s listen to what Joe Noel presented to Council and Council’s responses:

  • Ocala Exposed – Council Meeting 1 of 3 is Joe Noel’s presentation.
  • Ocala Exposed – Council Meeting 2 of 3 is Council’s Questions.
  • Ocala Exposed – Council Meeting 3 of 3 is Council recognizing our efforts to expose the truth.

 

“Sounds like the IT Assessment we got is flawed…” – Council President Musleh

“So we spent $180,000 and we are in no better position today than … before?” – Council Member

The new training that put their budget over was security and training to the new personnel because they terminated the only people that were certified in network security … Shawn and I spent $20,000+ EACH of our own money to be certified in network security and systems management. Why couldn’t they have leveraged our 15+ years each of experience of managing a network that has NEVER been down … 

“The reason that it was done was kind of flawed and political in nature and was done to prove a point and kind of skewed towards that ..” – Mayor Guinn

“We spent a whole lot of money to learn something that I could probably of told you with my limited IT experience…”- Council Member

“I hear what you are saying, if the premise of the original assessment was off-base or flawed politically, … you get what you got for $180,000 to basically… to say what you wanted it to say …” – Council Member

Facts from Joe Noel’s Update from the Coeur Business Group
and the “skewed” IT Audit:

  • Joe Noel is now the Assistant Director of Information Technologies ($95k??)
  • Before the IT Audit: 17 employees, personnel costs $1.5M and total base budget of $2.5M
  • After the IT Audit: 15 employees, personnel costs $3M and total base budget of just over $3M
  • Increase in the budget was due to the contract with their new managed services provider as well as the City’s new upcoming website redesign and the city-wide purchase of Office 365.
    • (By the way, your new website should not cost you any more than $10,000 to redesign and should cost you between $300 to $1500 per year for hosting… for as little as the core website does.)
  • Joe needs another $120k for backups? What? We were already building and redesigning the backup system to failover to the Sheriff’s office NOC…
  • Joe’s Bullets: They are setting up Virtual Desktops … The infrastructure team that was terminated was already spearheading, and well into testing, virtual desktops. We heard that the new IT employees tried taking our test VDI environment live and failed, obviously because the test environment was never meant to be the production environment.
  • Joe’s Bullets: They are decommissioning servers… Can you be anymore vague? The infrastructure team that was terminated was already decommissioning servers, as we have had our own cloud implemented where we had already decommissioned more than 125 physical servers. We also saved the City more than $2M one year by building our cloud infrastructure in lieu of the physical servers that OEU (Joe Noel) was insisting we buy for each project…
  • Joe’s top priorities according to his presentation:
    • Complete transition  of  all  our  2003 servers to 2008: We were already doing this and it should have been already completed no later than July 14th, 2015 … Running 2003 servers is a huge security risk. This is another thing that we were actively working on before the Coeur Group and Joe interrupted everything.  Windows Server 2003 extended support ended on July 14, 2015. Microsoft will no longer issue security updates for any version of Windows Server 2003. In the case of Windows Server 2003, antimalware support ends when the server becomes an unsupported product. Microsoft is not extending the issuance of antimalware protections for organizations not making the July 14 deadline. If he has not met this deadline, which obviously he hasn’t because it was in his presentation, he should be terminated for this alone. Congratulation, Joe has provided you a huge security risk, now live on your network!
    • Complete transition from Office 2007 to Office 365: Each year we researched Office365, when we started feeling it was becoming advantageous to the city, it was usually denied.
    • Complete rewrite of all policies and procedures: Horst had us delete all Policies and Procedures city-wide, “because if you had them you had to follow them … ” They are pretty much already written, but could use some editing. The old policies and procedures is still on file so this is minimal work and I can’t believe it is a bullet item in his presentation…
    • Create a comprehensive Disaster Recovery Plan: This was something we were already building with our backup ESX stacks (Cloud Server Network) at the Sheriff’s office as well as the the offsite backups and documentation, which ALL would have been done already, in time for 2015 Hurricane Season.  These offsite backups were designed so that the Active Directory would fail-over to the ESX servers and physical DNS server at the sheriff’s office, so if we were cut off from the rest of the network or from the entire world internet, our network would still be able function at the EOC for critical applications.
      Replace our current manual tape backup system with disk to disk system: This was something that we were already actively doing as well before Joe interuppted and it to would have been completed already.  We purchased the Dell Compellent SAN (Large Disk Array) give us more internal cloud storage for server images as well as to completely re-level our backups and to remove the tape backups for a disk to disk system. 

So again, Joe Noel is just taking credit for stuff that most of the terminated employees were already actively working on.  Each of these projects mentioned would have been fully implemented by now if the IT audit by Joe Noel and the Coeur Business Group wouldn’t have derailed everything.

The Infrastructure team that was terminated saved the City of Ocala almost $2M dollars on some OEU projects in a single year that Joe Noel was spearheading at Ocala Electric. (Ironically, we saved the City the cost of the entire IT Budget according to Joe’s numbers, by keeping Joe Noel in check) They wanted to buy physical servers for several projects. That’s how our current ESX Stack (Our in-house cloud network) came into existence. We researched technology and ended up buying the ESX system that we could use for, not only their projects, but also for other areas of the City that had needs for servers for their projects, including the active directory infrastructure.  We bought 6 servers that allowed us to run more than a 120+ virtual servers in our own in-house cloud. That $2M in savings alone, plus the other savings by going virtual, should have carried our network engineering expertise through this audit instead of just casting us out as it did.

 “It sounds like we made a mistake and now we are recovering from it.” – Council Member

President Musleh commented the IT assessment presented was flawed.  The City spent $180,000 and is in the same financial shape they were in prior to the assessment.  Mr. Zobler stated that areas of security and training were highlighted, and moving to cloud space was not a workable model in the City.

Unfortunately, your Infrastructure team, that was terminated, already knew all of this stuff, but the upper management (Matt Brower) would never listen because they had an agenda. They wouldn’t allow us to implement new security practices because they just didn’t like to have strong passwords or want to mess around with changing them every few months or so.  They wouldn’t allow us to migrate to Office 365 that last time because it cost to much, they wouldn’t allow us to purchase the FireEye Security Appliance because it cost to much…    For the price of the IT Audit, you could have bought 2 FireEye appliances, one for the City of Ocala and one for OPD and still saved money.

And migrating to Office365 was looked at for several years in a row before the audit by the infrastructure team, as we were wanting to make the move when it became as advantageous as possible cost-wise for the City’s budget.

Apparently, paying $180,000 for an unethical and politically skewed audit, terminating 4 good and highly critical employees to the network infrastructure, and installing an unqualified Joe Noel into a management position in IT was more cost effective to Joe Noel and Brower …

Remember the billing system issues?

Right after Joe terminated the 4 employees, one of them being the DBA that used to protect the billing system, an untested patch was run into the production billing system that double charged all customers… How much did that cost the city to fix? That was caused because of Joe Noel’s and Brower’s actions.

This is an excerpt from my email requesting funds for the FireEye Security Appliance:

In the three week we have had this device, we were able to identify several malware vulnerabilities that got through our defenses. I will say that I was VERY impressed to know that our defenses are protecting us very well, but we did have some items get through. One malware item alone was from a PDF; this item changed more than 12,160 files and more than 180 registry keys on the target PC. It also performed Data Theft, by attacking the auto-complete passwords from the installed browser and attempted to send those out. If a user was saving their credentials in their web browser for EDEN, or the Intranet or any other of our web applications, this user’s credentials could have been stolen.

Another Malware item we caught delivered a rootkit and made more than 170 file and registry changes to that target PC. This was a ZERO DAY malware, as when we went to do research on it, only one anti-virus company had it listed and they listed it about 15 minutes before we witnessed it on one of our PCs. The next day, ALL of the anti-virus companies had it listed. This is the best example of successfully finding and mitigating a ZERODay threat on our network.

One of the more severe Malwares we found just yesterday called Exploit.Kit.Darkleech. There has been a recent rise in drive-by attacks from the Darkleech attack campaign that’s been using compromised servers and malicious Apache modules to launch drive-by attacks that target known browser vulnerabilities. They use JavaScript to attack anyone browsing the website. If the attack is successful, the malware redirects the browser to another malicious website, where a crimeware toolkit attempts to further compromise the PC. This can be from legitimate websites if they are compromised.

These two PCs that were infected with Darkleech. These malware infected PCs, could have sat for weeks or months on our network before we truly realized what was happening because we just would have never known.

Using the data from the FireEye on the items that got through, we were able to tweak our firewall further to help protect us against them in the future. The FireEye would effectively block any outbound traffic if setup inline, where malware would not be able to communicate with their command and control servers.

I feel it’s imperative that we implement this tool in order to protect ourselves from malware and other events that traditional signature-based security just won’t protect you from, especially ZERO DAY attacks. – Tommy Thomas, City of Ocala’s previous Microsoft Certified Systems Engineer.

 

City Council, please understand that your infrastructure team, that was terminated before you all were even given the chance to hear the Coeur Business Group’s findings for the first time, or even allow us to give you sound advice from a technical point of view, had already built and managed our own cloud network infrastructure for the last 8+ years or so and had already decommissioned more than 120 physical servers and already started implementing the Virtual Desktops…

We owned our own cloud, the network has not gone down in more that 12 to 15 years that we were managing it. We adhered to Microsoft’s Best Practices and forged a secure network to be proud of.   None of our efforts under Horst or Brower were ever allowed to see the light of day.  Know that Joe Noel was working closely with Brower, and he’s always fought to be over IT even when he was just a GIS Tech, you have now handed your entire system to someone unethical and untrustworthy.

The direction you are going in, is the same direction that we had been trying to take it for years.  We were right-sourcing as we went when allowed, while maintaining a 99.999% up-time, while managing all security and infrastructure with only 3 or 4 of us.

So why is Joe Noel still working for the city after Joe Noel intentionally allowed Council to be misled in order to get his agenda pushed through, as well as to cover up his own failed systems that were under his control? Where do you go from here and how do you continue to trust anything Joe Noel says, or anyone else that supports his agenda?