Posts

City of Ocala needs new leadership in wake of hostile I.T. takeover!

The Coeur Business Group’s IT Audit for the City of Ocala was more of a hostile I.T. take-over by Matt Brower & Joe Noel.

UPDATE 11/2014: Matt Brower was let go as the City Manager for the City of Ocala in October 2014. John Zobler will be taking his place. I’m hoping that he will be allowed to truly clean house and make the City of Ocala a great place to work once again.  John Zobler was one of the few managers, through the years, that I truly respected. Joe Noel is still the Chief Technology Officer. His resume was added below.

Please know that slander or libel is not intended in any of this document, this is our opinion and should be taken as such.

The I.T. audit by the Coeur Business Group came with mixed emotions from most. However, we (the Infrastructure team) were excited. After all those years of building our network from the ground up, we were actually looking forward to the IT Audit by the Coeur Business Group. We were told that the Coeur Business Group’s IT audit would be unbiased. If so, it would give us the leverage we badly needed against all the upper management who just wouldn’t listen to reason through the years.  Turned out it just wasn’t so and there was a clear agenda in the end.

We remember our first meeting with Dan McLane, Coeur Business Group Inc.’s senior partner. McLane gruffly said to us, “If you have any skeletons in the closet we are going to find them.” Shawn Hoff, our Infrastructure Manager, quickly replied “You don’t have to look in the closet, the corpses are all over the floor.”

We freely gave the Coeur Business Group everything they asked for, and much more, in hopes they were there to truly help us make positive changes. A change where the department managers and assistant city managers (ACM’s) would have to actually listen and be held responsible for not adhering to common sense security practices that we had been trying to establish, or even just to listen to our advice on costly applications such as Eden or the Cogsdale application.  This was a change that we desperately needed to properly keep the City of Ocala’s network secure.

The Coeur Business Group used all of our information against us
as their own personal discovery.

One basic example: The infrastructure team, for many years, tried to enforce strong passwords, which is one of the most easiest things to do in network security.  The previous City Manager, Ricky Horst, was against strong passwords and had his assistant city managers quickly call us to reverse the changes within 4 hours.  We tried again with Matt Brower, but he called Shawn Hoff, the Infrastructure Manager, within 24 hours and again made us turn off strong passwords, because he, and other Assistant City Managers, did not like inconvenience of the security change. The Coeur Business Group used this as one of their security bullets against us even though we fully pointed out that it wasn’t IT which was responsible for the poor choice.

If we couldn’t get strong passwords (a simple and free security setting) approved through management, how in the world would we be able to make anything secure, especially for larger items they could never understand or comprehend.  A good example would be the FireEye appliance, which we requested for the past few years in a row, or choices in major applications such as Eden and Cogsdale applications. (Both applications were strongly not recommended by the information technology department.)

The FireEye Network Threat Prevention Platform stops network-based attacks missed by traditional technologies, such as next-generation firewalls (NGFW), IPS, AV, and Web gateways. Interestingly enough, the recent Target hack could have been avoided as they had almost a million dollars in FireEye equipment but did not have it properly setup. They lost 40 million credit card numbers. Read More …

A good example of the Coeur Group’s incompetency was during their initial scan of our Active Directory, when they alerted me that that they found “serious problems” and that my Active Directory was showing a lot of errors and most likely not functioning properly.  I asked them to show me what they were talking about, as we had just corrected a replication issue with Microsoft Support after adding a new domain controller and upgrading the network infrastructure. Microsoft had just told us that our Active Directory was setup and running optimally.  The errors they found:

  • NCSecDesc

    According to Microsoft, If you do not plan to add a read-only domain controller (RODC) to the forest, you can disregard the failed NCSecDesc error.

  • Invalid Service Startup Type: W32Time

    Also according to Microsoft: This is an expected behavior as the W32Time service is changed to a manual startup service as default in Windows Server 2008 release two (R2).  Since we use Domain Time II to synchronize the time on the server network this is also expected.

These two errors are very common and I was very surprised that they tried to say they were serious problems. They very quietly agreed that the Active Directory ‘errors’ were, in fact, not really errors at all.

Did you know the official City of Ocala logo was a $40 font that cost taxpayers almost $10,000?  If they listened to IT, the logo would have only cost $40.  I had to follow up to make sure the design company actually paid for the font, which they hadn’t.  Did you know that your ACM’s and City Manager claim it’s trademarked, but actually the tagline “Find Your Place” is trademarked already. How much would it cost if you have to change all that signage if someone decides to protect their trademark and wins?

Let’s discuss the physical security of the building for a moment which the Coeur Group so kindly pointed out and blamed IT.  The security swipe badges and doors were all done by, and the sole responsibility of, Joe Noel. Joe was adamant about not letting IT have any part of, or access to, the equipment or software. The door locks were a huge failure in our opinion as the doors were often left unlocked all night, including the ones that lead to the server room in City Hall.  We even had a homeless woman living in the main lobby for an unknown amount of time, long enough to have a hidden panel to store her stuff in the wall just outside of the Clerk office.  In my opinion, Joe should have been forced to retire over the door security failures which potentially endangered City of Ocala employees as well as the integrity of City Hall, the server room and the network.  IT and Telecom should have been in charge of the door security from the start.

As far as the network ports in conference rooms, just like the strong passwords, none of the directors or assistance city managers would ever support our decisions to tighten security.  In 2011, it was really aggravating as our very own future IT Director, Winsome Jacobs, (application manager at the time) was letting people (non-employees) hang out after hours in the IT break room.  As far as I’m concerned, she should have been fired for such a gross lack of security, not only for the data on our network, but for the safety of other employees.

We also asked to reinforce the IT Server Room in City Hall as the panels on the bottom of the windows are nothing more than plywood which can be cut by cordless saw, which would give complete access to that server room.  We ended up giving up on that request as we knew we were moving everything to the Telecommunications Network Operations Center (NOC).

Since the City of Ocala terminated three of the four infrastructure employees and one help desk employee, (two veterans, both with more than 14 years with the city, and one prior-service with 24 years at the city) we have heard that project managers are being trained. This is funny to us as most of our current project managers can’t troubleshoot the simplest of problems for their projects they are there to manage.  One example was with the CSMS application and it’s “Call Stack Errors”.  After looking into the “Call Stack Error” for that project manager and their team, with a simple 5-minute Google search, I pinpointed bad coding.  The vendor ended up confirming the problem was from their bad coding as well. It’s frustrating from a technical and professional point of view, that the project manager, nor their team, did any research for the error in the months they were complaining about this problem.  The big issue is that they just don’t have technical trouble-shooting skills, like employees that actually hold IT certifications.  Please know that most project managers were not even in IT, but at Ocala Electric.

The Coeur Business Group stated that the City of Ocala should be training good people.  Shawn Hoff and I spent more than $20,000 EACH, of our own money, getting the certifications and knowledge needed to run and protect a network. I have my Microsoft Certified Engineer (MCSE, MCSA & MCP), Certified Ethical Hacker (C|EH), Network+ and Security+, as well as many classes for CCNA.  I don’t see why you would ever get rid of good employees that paid for their own education at no cost to the taxpayers.

I heard that Matt Brower recently listed getting rid of 4 employees from IT as an ACCOMPLISHMENT of his.  How would that be an accomplishment unless he really was working out people purely motivated by pension maturity?

IT Director Winsome Jacobs and Tom Jenkins are both very nice people but not remotely qualified for the positions they hold. Winsome’s knowledge of physical security mentioned above, as well as the lack of knowledge on basic email security, use of domain credentials or even just malware is appalling.  She was often putting important application credentials into unsecured emails.  After Shawn was laid off, I had to explain to Winsome that even if you disable a user’s network credentials, if you don’t change the other credentials they know as well, they can still log in if they wanted to. She had a difficult time understanding something so basic and how that was even possible. An IT director that doesn’t have a grasp on the basics of network security, or networks in general, puts the City of Ocala at risk. If she was responsible for IT as a director, why was she kept on after the audit and given almost a $5,000 raise?  Seems like the director should have been let go before any personnel.

Tom Jenkins also has no concept of basic security or networking in our opinion.  A good example of this was when he asked us to reset the Mayor’s password to the Financial System.  The requested password was sent via an un-secured plain-text email that plainly associates the mayor’s name and job title, with the password; as well as references the application he has access to, which houses a lot of sensitive information. Not to mention, more than 10 people now knew his password since it was requested via email to our support department via theirs. That alone is a serious breach of security for an important city official.  Again, this was done by an IT director, not a junior help desk person.

Another example was with network time where Tom and Winsome were sending emails back and forth seemingly trying to show each other there was a time issue. Unfortunately, with Tom, it was always a problem that we had to troubleshoot with no information or access. Tom’s email said they ‘think’ they match National Institute of Standards and Technology (NIST). It’s pretty easy to confirm if you do or do not, so there should be no ‘thinking’ involved. You either do; or you don’t. 

Another good example of working with Tom Jenkins at OPD was with an email address forwarder they deleted on their network. They are almost always random shots fired over the bow with no information. For this particular incident, Tom ended up recreating the deleted email forwarder on their end so they could again disseminate email to the Eden users which is why it was created in the first place. Not sure why they deleted it at all. It’s not that we didn’t want to help him when he called, he just made it really difficult to do so, as he doesn’t have a firm grasp on troubleshooting his own network in my opinion.  After years of dealing with him, you kind of just give up and pray that someone who knows how to truly manage a network takes over OPD’s network. He’s very fortunate that he has vendors he can call to fix his firewalls and everything else.

Let’s talk about Joe Noel. It was well known that Joe always tried going around IT.  He was one of the main ones that spearheaded projects and would not involve IT at all, mostly when Brower was the Director of Ocala Electric.  One project, Shawn Hoff was able to save the City hundreds of thousands of dollars by ordering the proper hardware vs the hardware they insisted upon.  The Cogsdale application was way over budget and 2+ years past deadline as well. This is a good example of an application that was dropped into IT’s lap, after it started failing.  IT highly recommended against it once we found out it was being purchased.  Jeff Bixler, one of the infrastructure team that was laid off, was always very careful when updating this application and the other applications he had to oversee.  The issue where residents were double-charged was purely because your ‘project managers’ were not more cautious of pushing in patches, literally within a month of Bixler be laid off.  I would be curious as to know how much that cost to get fixed?

From what we understand, Joe Noel has no technical background to be a Chief Technology Officer.  Before being a GIS Tech with the City of Ocala Electric Utility, he was in the food (pizza) industry from what we understand.  We think this is why most of Infrastructure team was let go early, BEFORE the Coeur Business Group presented to their findings to Council.  The IT Infrastructure Team, consisting of Shawn Hoff, Jeff Bixler and myself were the only ones in IT that could recognize Joe Noel’s lack of knowledge and pinpoint major flaws and errors in the decision processes that had been made in the past and that will be made in the future. Tom Jenkins and Winsome Jacobs, in their own lack of knowledge, will facilitate Joe Noel’s poor and costly decisions that will be made with the tax payer’s dollars.

I would also ask the City Council to have an auditor examine Joe Noel’s college credits that he claimed to have, in order to find out if the rumors are true that he allegedly falsified his job application with the City of Ocala.  If so, this is definitely not the person you want in a management position taking almost $100,000 per year from the taxpayers.

UPDATE 11/11/2014: Here’s Joe Noel’s resume from a public records request through the City of Ocala. According to the public records request, when asked about the validity of his education, they responded with: “On January 16, 2008 a request was sent to North Metro Tech in an attempt to verify Joseph Noel’s education. No written documentation was ever received back; but a phone call was received from Ms. Weeks stating that Joseph Noel’s education is more of a certificate than a degree.”Papa-Johns-Logo-and-Pizza

After he got his ‘certification’, he then went on to work at Papa Johns; a pizza shop.

So, he basically stretched the truth about his ‘certification’ by calling it a degree, or maybe he completely believed it was an actual degree. While he was obtaining this certification and making pizzas; two of the 14+ year City Employees, that were recently laid off, were busy serving their country, one in the United States Navy and the other in the United States Coast Guard.

While he was busy depending on this one ‘certification’ for his entire career, more or less; we were busy constantly educating ourselves by getting our IT-related certifications such as; Microsoft Certified Professional (MCP), Microsoft Certified Systems Administrator (MCSA), Microsoft Certified System Engineer (MCSE) as well as Network+, Security+ and even Certified Ethical Hacker (C|EH) and a host of other classes, to a total cost of more than $20,000 each.

Joe Noel makes a $95,000 yearly salary with the city, holds no degrees or certifications to qualify as a Chief Technology Officer and, from past experience, is not capable of making sound technology-driven decisions.

These are small simple examples, but they speak volumes of the people that are now entrusted to manage the City of Ocala’s technology.

This leads us all back to Matt Brower. His decision making processes are flawed.  Instead of ACM’s and a Chief Technology Officer, how about a highly qualified IT Director? Why not give that IT Director, and other directors, a direct line of communication with a solid City Manager. Paul Nugent only needed one Assistant City Manager …  Then you rely on that IT Director, and their team, leveraging their knowledge to help move the City of Ocala forward by right-sourcing gradually, rather than shotgun-style outsourcing.

Like Joe Noel and Winsome Jacobs, Shawn Hoff applied for the IT director’s position as well.  In his interviews, he basically told them all about how we need to right-source, not outsource, and how we could leverage newer cloud technologies.  We didn’t need to spend $180,000 to tell the City Manager what we already knew.  Shawn and his infrastructure team saved the City of Ocala almost $2.5 million by buying and building our own cloud infrastructure instead of listening to project managers, who wanted to to buy physical servers for each project.  We bought and built a cloud infrastructure with about 10 physical Dell servers using VMWare’s ESX which now host more than 100 virtual servers.  The City of Ocala’s network is hosted in its own cloud utilizing just 10 physical servers, where all of the hardware is owned by the City of Ocala.  We were always proud when we could identify where we were able to save the City money that covered all of our salaries. It was always the challenge to do so, all while leveraging cutting edge technology to bring a higher level of security and date integrity.

When the Coeur Group used all the information we gave them against us, we immediately came to the conclusion that the $180,000 spent on the audit was spent to place Joe Noel over IT. Almost like a hostile take-over at the expense of the taxpayers. Joe Noel has wanted to be over IT for years. Unfortunately for him, all his projects were always over budget and the technology haphazard. We kept him in check for many years and saved the City of Ocala hundreds of thousands of dollars in doing so each time. The only thing we can figure is that Joe Noel and Matt Brower initiated the IT Audit to nullify Sandra Wilson’s choice of IT Director, and to install Joe Noel, with no known credentials or technology experience, into the position of Chief Technology Officer; no matter what the costs to network/data integrity or the security of our network.  All should be fired for playing games and wasting tax payer’s dollars and putting the City of Ocala’s data and network at such risk.

We are more than happy for the City of Ocala Council to change our “Workforce Reduction” to a short “Administrative Leave” so we can come back in, reclaim and protect our network.  Our only stipulation would be that Matt Brower be let go as City Manager; Joe Noel, Tom Jenkins & Winsome Jacobs be made project managers or forced into retirement/released. Assistant City Managers positions, as well as the Chief Technology Officer position, should all be removed to save the City of Ocala money.  Shawn Hoff and I should be placed on the board that is responsible for hiring your next IT Director, if the workforce reduction is reversed.

The word is that recently added managers are delegating all their work down and it’s piling up, setting up good junior employees for failure by more poorly chosen managers. This seems to be how the entire City is being run.  Since Brower/Horst and the ACM’s we now have 3  unions which should tell you something. None of them are making the employees feel comfortable, or even trusted.  Most of these issues in this article are with IT, but it goes much deeper than just us. The City of Ocala needs to support the employees. Managers should be chosen that are not only knowledgeable, but respected by their peers. This alone would foster a better working environment from the ground up; an environment that would probably provide a more rounded home-grown City Manager choice that knows the local area, and employees, as well as appreciates and leverages local existing businesses.

Council needs to act fast to end the nepotism that leads to unqualified people making crucial decisions that cost tax payers ’s dollars. The Council needs to uphold the City of Ocala’s mission statement: The City of Ocala provides fiscally responsible services consistent with the community’s current and future expectations.

Thank you,

Tommy Thomas, MCP, MCSA, Network+, Security+, C|EH, MCSE
Network Systems Administrator -::- Webmaster -::- Public Affairs Specialist (U.S.C.G.) -::- Photojournalist

IT Increases/Promotions after May 19, 2014
Effective DateTransfer Type CodeEmployee NameOld Position TitleNew Position TitleOld Hourly RateNew Hourly RateOld SalaryNew Salary
5/12/2014promotionNOEL, JOSEPH PSUPV, ELEC ENG RESOURCE MGMNTDIRECTOR OF PROJECT MGMT/CTO$40.97$45.67$85,237$95,000
5/12/2014reclassJACOBS, WINSOME FDIRECTOR OF INFORMATION TECHNOLOGY/ CIODIRECTOR OF INFORMATION TECHNOLOGY/ CIO$44.23$45.67$92.000$95,000
6/4/2014promotionHUNT, ADAMSAFETY/RISK COORDINATORCERTIFIED PROJECT MANAGER$24.03$34.85$50,000$72,500