> CUNY Central was so eager to have a centralized MIS tool to use for its own centralizing, corporatizing agenda, that it totally ignored the implications of the Oracle "configure-only" limitation: business processes would have to be made to fit Oracle, not vice versa. Capabilities that we now have will vanish. The staff, the faculty, the students would just have to "adjust" (the technical term being "suck it up").
This is an interesting perspective. From what I've seen / heard from others, it's generally better to adapt processes to the off-the-shelf tool than to try and customize or build from scratch to accommodate your bespoke processes (especially in the business operations realms). For one, the organization is likely less unique than you think, and those bespoke processes are as often a function of some early employee's preference as it is a genuinely good reason. For another, customizing software is not just a one time cost, since every subsequent update / upgrade is likely to require additional work (or at least testing). And finally, in most cases the closer you are to a vanilla, standard process, the more likely you are to stay in compliance with local laws and regulations.
Though I suppose it's possible that the imbalance is just due to the fact that it is much harder to quantify the costs of using a suboptimal (for you) process than it is to look at the procurement contract for a custom solution.
I've had the opposite experience. Any attempt to adapt processes to an off-the-shell tool have always been a subpar experience for everyone. I've brought a few of these in house as fully custom software and the end result has been a better user experience and faster changes. If we can buy something that fits the need, we will. But if we can't, we build. And we build a fair bit.
I disagree with "the organization is likely less unique than you think". If you're big enough you will have unique requirements that nobody else has. I'm in the middle of that now in a project to install some industry standard software that runs the whole business and we have to customize and add custom integration into it. I wouldn't want to build software this complex in house but if I was given the resources and tasked to do it, I could, and it would be better.
Scenario 1: You're doing something that every other business is doing. E.g. ERP/accounting, sales, contact center, etc.
Scenario 2: You're doing something few other businesses are doing. E.g. your actual customer business, creative, etc.
(1) is amenable to making your process fit software, to good results. (2) is usually a train wreck.
Unfortunately, figuring out if your thing is scenario 1 or 2 is non-trivial.
Canonical example: EMR/EHR systems in healthcare. You think they'd be the same... but actually there are so many integrations with other systems and/or different sorts of specialists, that a real world implementation has substantial functionality gaps (papered over with custom work).
My impression is that most people don't understand just how awful most commercial business software actually is.
One thing our business does that every other business does is vacation and overtime tracking. We have a custom in house application for that and we've yet to find a commercial replacement that is, in anyway, half decent. For most Payroll/HR systems, this is merely an add-on feature and doesn't get much attention.
For overtime, integration with our financial system allows overtime to be charged to the correct files and this is something that nobody does (or does well). Probably doing just this little bit makes this project pay for itself.
Commercial business software manufacturers are isolated from the users and in a way isolated from consequences as long as they fulfil the contractual obligations (which practically never has a 'make the users happy' stipulation).
Beside the lack of attention, you also have gargantuan legal requirements you need to integrate. Which change all the time. Sometimes a few per country.
One advantage of building in house is that you're only building for your own company. This is significantly less work than building commercial software for multiple clients (which I have also done). I can't overstate how much less work this is and how much of a better experience it can be for users.
As an example, for calculating annual vacation entitlement, we have some pretty complicated rules. But every company in every country has their own set of rules so most HR software doesn't bother calculating it -- you just figure it out manually and input it every year for every employee. But because we just have one "client" our rules are just code that we can change as needed and can arbitrarily use whatever information we have. This saves HR a ton of manual work all the time. But this only works because it only needs to be one set of hard-coded rules.
> has their own set of rules so most HR software doesn't bother calculating it -- you just figure it out manually and input it every year for every employee
Beg to disagree. This is the complexity that large ERP firms handle and why Oracle, SalesForce, etc are expensive to implement. They figure out the commonality (if any) and build for it. Then they add on features specific to countries they target and then they add the ability to configure for your own situation (to a certain level).
PeopleSoft did this for Payroll and workforce administration which is part of how they cornered the market for HCM.
The problem when building it yourself is that this is usually done by "generic" developers who discover edge cases (often together with the requester) that threaten the whole model.
A company doing payroll for us ("us" being a multinational company) asked for a "typical payroll" to start with. Fortunately we had experienced people on the pay side who discarded the company because of this question (for one they should know, and for two they should know that "typical" will cover maybe 60% of the cases -- I thought that this was an exaggeration until I discovered the reality of calculating pay in France)
A good company specializing in "pay" or "vacation" (which are very regulated over here) will know the "typical" case and the edge cases.
I think most moves to EMRs were insufficiently disruptive, e.g., electronic orders recapitulate the old paper orders without using the opportunity to reduce ambiguity and insert constraints to prevent common errors.
The problem is that the paper form is the interface, because the ecosystem was designed around it.
10 years ago or so, I asked a health insurance company why a specific digital form could only list up to 16 diagnostic codes (otherwise a duplicate with the additional needed to be created).
They thought about the question for a second, then said that's how many were on the paper form the digital system had been created from (35+ years ago).
> If you're big enough you will have unique requirements that nobody else has.
Definitely agree, and that is very interesting to hear. I only mean to speak to my experience, and what I saw in a lot of cases was unique requirements that were due to the organizational equivalent of tech debt (ie things like "our books are organized in this unique way because we acquired this other company a many years back but kept their stuff separate because it was the path of least resistance at the time").
That’s a legit issue. What’s the value of changing a deep seated process?
I have definitely seen examples on both sides of that question. Especially in a place like a public university with multiple collective bargaining agreements. The unions aren’t going to accept significant change without some sort of cost.
Typically, since processes are built around the system, nobody understands the actual business needs.
> If you're big enough you will have unique requirements that nobody else has.
the problem is when EVERY process is that way.
yes, each business has unique aspect to their processes, but when every process is heavily customized by people who have no business designing processes & applications, organizations start to hamstring their flexibility and scalability.
Its like hand-making a Ferrari with completely custom parts when what you need is a Toyota Camry. If you aren't gonna race with it, its a waste of money.
I have been working in the ERP space for over a decade. and almost ALL customers have no criteria for when to customize or keep a customization. They don't do any cost benefit analysis or any strategic planning beyond building a customized tool for the thing immediately in front of their nose.
Personally I hate software customizations. I prefer to build from scratch using proper software development tools than use whatever piss-poor customization system commercial software typically provides.
I recently sold my director on building some software from scratch rather than try to combine two commercial products together for some Frankenstiened solution. The motivation for Frankenstiening it is that we are paying for both services so we should use them (and at least one is very customizable). Only after 6 months of failing to get these systems to play together in an acceptable way, I finally asked a team member to spend 3 days building a mock up of a new app. From there it was quickly approved and we hope to roll out next month. I only wish I had done that sooner; we would have had way more time for development.
Exactly this! I recall a large survey of SAP deployment projects in the late 1990s. By far the most successful consultancy, out of Chicago I think it was, had it written in their contracts "you'll change your business processes to match SAP; not SAP to match your existing business processes" (more or less). By turning away clients who could not accept that, they had happy clients, happy employees (little burn out), and no runaway costly never-ending death march projects.
There's a bit of selection bias going on there though. The reality is that SAP and similar products are designed for a business that works a certain way, and so obviously businesses that fit that profile are most likely to get value from using the tool. However, there's a reason other businesses don't work that way, and often retooling to work the SAP would be a net negative. Sometimes retooling SAP to fit the business is also a net negative, in which case the right choice is just to not use SAP, but I've certainly observed cases where there was a benefit from refactoring the tool to fit the business.
On "retooling SAP": SAP deliver their systems with all source code and dev platform included, and that may help convince some customers to go for SAP.
However, those that embark on deviating from the well-trodden path are going to be in trouble soon: after every update, potential changes made need to be re-done or edited or at least tested. So as the parent suggests it's really better to adjust the business process if you can.
> So as the parent suggests it's really better to adjust the business process if you can.
That's another way of saying that there are serious trade-offs going that way that need to be justified... which may also be true for the path of adjusting the business process.
There's a ton of variety out there in the real world such that you'll always find a few businesses that match almost any scenario you can imagine. So, yes, for some business doing a process a certain way is a competitive differentiator or advantage ... or even a necessity for their particular industry. For these businesses banging the SAP square peg into their special BP round hole is worth attempting. Even better might be just building their own custom round peg. I'm suggesting that many (not all) businesses are doing a BP a certain way for no compelling reason whatsoever other than that's the way they (almost randomly) picked to do it decades ago and could easily change to a standard practice without loss of competitiveness (maybe even gain by focusing on what does).
> There's a bit of selection bias going on there though. The reality is that SAP and similar products are designed for a business that works a certain way
ERP products are designed following "standard" or "best" practices/processes. It's common to see companies first contract a consulting company to "re-design" their processes before they then try to implement an ERP system.
...and yet there are all kinds of segments where customized tooling is more the norm than otherwise. It just depends on whether the deviation from the norm is a competitive disadvantage or a competitive advantage. There are a lot businesses where in at least one case, it is an advantage.
That's just days it's visible to be selective in your customers, not that all businesses have the same processes or would be better off not customizing.
> From what I've seen / heard from others, it's generally better to adapt processes to the off-the-shelf tool
I have never seen this work even once. I've actually built entire businesses on the concept that people are impossible to change, but software is easy to change.
Customizing ERPs is where consulting firms make money but the ERP vendors themselves advise against this because it becomes expensive maintaining the customizations as new versions of the software and more features are released.
Did you use much off the shelf software? I mean, I'm guessing you used databases and frameworks, but were the business process and UI components off the shelf or custom?
I use as much off-the-shelf as possible. But there is real work involved in adapting software to a company's culture.
If you do it right, 99% of the work has already been done for you by an existing SaaS or FOSS project, and "all" you have to do is customize it. And generally this doesn't mean "change the config" but rather assemble different building blocks into the right shape.
This can still be a multi-year project, even when you're writing almost no code. It takes a long time to understand an existing company's culture, processes, and needs.
Makes sense, this is just my personal experience so it's always interesting to hear what others have seen. I recently learned about Mechanical Orchard[1], which seems to have the same thesis (better to update legacy custom software with a modern custom solution rather than migrate to a modern off-the-shelf solution).
My employer used Peoplesoft, in the days before Oracle bought it. Of course, you can't set up Peoplesoft on your own, you bring in consultants. When I reviewed my notes from all the pitches, I found that they all had said that we should modify our processes to suit Peoplesoft. I guess we did--I was new there.
I would also say that painful, unduly costly implementations seem pretty standard for ERP products. I have heard such stories about SAP.
The problem is, the business process shipping with Oracle may be grossly inefficient. Or worse, the former Siebel module and its integration with a former PeopleSoft module may be replaced by some new thing with yet another business process.
In any case, CUNY would likely save $300M hiring clerks armed with excel and paper forms.
Seems like a lot of money wasted, mostly to advance a “lowercase p” political dispute to consolidate HR functions with a questionable benefit.
Now you have two problems: rolling out the new system and changing the business process. And you'll solve them simultaneously, and if anything goes wrong, you'll roll both back...
Business processes change all the time. Often you are better off changing - if you are more like everyone else you can hire someone who knows your processes and they are useful without a lot of training. So short term costs, but long term better.
I work on bespoke ERP software, it's not an undertaking to be taken lightly, but you are missing a key point.
Every process that doesn't your software doesn't accommodate becomes a manual process. There is an employee in a sibling department, using other software, who's full time job doing the work of one of my applications in excel.
I know hating on Oracle is en vogue, but I struggle to believe this $600M figure.
a. I use to work in this space, even a $6M deal would be massive (let alone something 100x bigger).
b. The ENTIRE cuny budget in 2013 was only $2B [0]. This isn't their IT budget, this is literally the entire budget to run the entire university system across multiple campus, faculty, buildings, etc.
c. Because Higher Ed is known to be so cheap, especially in the late 00s - big tech companies charged accordingly (which was at a massive discount relative to most other accounts).
d. even if this $600M figure was an aggregated figure over multiple years, staffing and auxiliary costs - it still wouldn't come close to this figure.
e. an expenditure this large would definitely be called out in CUNY annual financial reports, and I can't seem to find any reference to it.
EDIT: I did find reference last year to a $175M funding request to migrate from on-prem PeopleSoft (Oracle) to cloud.
Though, what I've historically seen is that only 10-20% of the funding request actually go to the software vendor. Organizations typically add 3-5x additional to either "pad" their request (in the event it doesn't get 'fully funded') and/or this is an opportunity for the university to higher for a bunch of roles they wouldn't have been able to get funded in the first place - so lots of things get buried in these numbers.
Lastly, the figures are also typically multi-year. Like 5-years being asked up front for approval.
Said differently, it wouln't surprise me if the true annual migration cost from onprem to cloud PeopleSoft might only be $10-20M
I cannot imagine any college paying 600 million for such a thing. I suspect the writer is very much mistaken on the price and a bit paranoid on the College's motives. Perhaps it is 600 million over 10+ years.
Why they would ever go with Oracle is another mystery as there are a number of vendors who specialize in this type of software for colleges (some of it good, most of it bad), so it would be foolish to explore a solution from Oracle that needs such a high level of customization to fit the needs of a University.
You are likely correct. My best guess is they hired someone in IT from outside of higher education that had extensive experience with Oracle and had deployed it successfully in the past and so it was always going to be Oracle even though it was a poor fit.
We don't know the true scope of this, but apparently $600m was low balling in this story:
>The negotiations that were the run-up to the purchase of CUNYFirst were a travesty. The project required an expenditure of up to a billion dollars to do it right. CUNY Central offered far less. All but one of the bidders dropped out as a result: the project could not be done properly with what CUNY offered. Oracle-PeopleSoft did not drop out. However they warned CUNY that for that level of funding, they could not, would not CUSTOMIZE: they would only CONFIGURE.
I do agree that it sounds like this was some 10+ year project with heaps of support. If this account it true, they went with Oracle because they were the cheapest (and were fine with the compromise of no configuration to hit that goal).
If that hunch is true, I can see some $600k-$1m/year average for support of an entire chain of universities (remember that CUNY is a systetem of 10 colleges and more CC's like a UC, not just one campus) being peanuts that no one wanted to touch.
Seems they got what they paid for:
>The actual cost far exceeds the $600 million dollars that go to Oracle. Because processes are now much more inefficient, more people have to be hired to do tasks that were formerly automated.
> - The interface is laughable: It looks like an early-90s update of 3270 bi-synch technology. Web 2.0? Ha. Not even Web 1.0.
> - Because CUNY wouldn't pay for customization, we had to renumber our courses. This is just one of many, less visible to faculty, changes that CUNYFirst has forced.
absolutely not surprising as someone who's used Peoplesoft. That stuff felt ancient even in 2003. to sign onto that 10 years afterwards for a very stagnant software is just insanity.
-----
now of course I'm taking this at face value. It could be exaggerated or from a biased source. But in my biased opinion, who in the last 15 years isn't biased against Oracle's software? they will probably literally die out as their clients age out of the job market.
I once submitted an RFP response for what was literally a CRUD app for a government agency. Medium-ish (to me) traffic. Needed to be rock-solid. Needed to be developed to spec, hosted, and supported (including tech support) for 5 years. I bid a little over $2million (I personally would have made about $500k of that). After a bit, the RFP was withdrawn and given to a sole-source provider for $100million, with the excuse that it was so complicated (HN would laugh at this if you saw the spec) that only one company could do it. So, that company gets it. Outsources everything to contractors in Pakistan. Site is constantly glitchy. And this next part is not a lie (and was def. not part of the spec) -- it only worked during business hours (m-f, 8am-5pm)! However the users of the site had to do work every day due to legal deadline requirements. They just had to set on the data until the next workday morning. I suspect that the provider spent about $1million or less for the entire contract and just kept the rest. So that is how govt. contracts can go.
With government bureaucrats with no penalty for failure and no expertise in software development guiding the design? And “lowest bidder” contractors to implement whatever shitty design the bureaucrats came up with?
That's my dream. One day I hope to recommend someone spend more money on me because I suck at what I do. So long as it's not their money, they should be quite willing.
>Though, what I've historically seen is that only 10-20% of the funding request actually go to the software vendor. Organizations typically add 3-5x additional to either "pad" their request (in the event it doesn't get 'fully funded') and/or this is an opportunity for the university to higher for a bunch of roles they wouldn't have been able to get funded in the first place - so lots of things get buried in these numbers.
As someone working on implementing an Oracle solution for a large bureaucratic MNC this is also true. The padding often is 3x-5x with 5 year run rate the funding asks can be crazy. For people who are just looking at numbers it can seem crazy. But for anyone who is implementing it or renewing contracts they know the real numbers.
The migration costs though might be higher than $10-20M. Contractor costs can be crazy too. Sometimes costing multiples of the software costs.
In my current practice I do notice that the vendors - both software and third party implementation partners - have a chummy relationship with a decision maker. This creates a lot of misaligned incentives when it comes to "company's" money.
Not quite true, especially not for the higher end research universities.
The University of Washington is projected to spend $2B on modernizing IT infrastructure (which does seem like a reasonable amount).
In that number though is $340M to update it's accounting system. Not quite directly comparable to $600M for HR, but not completely perpendicular either
Plus one on this. I've almost never seen a 9 figure deal in my life, and the only ones I've seen are massive core infra level deals by mega-corps. I absolutely cannot imagine CUNY spending more than most F50s spend on HRMs.
Edit:
I did some digging into the Tender [0] and it looks like it was $300M total across all 25 universities and colleges in the CUNY system along with the CUNY administrative portion itself and it was a full bundle of HRM, ERP, On-Prem Compute Resources, and headcount (PS and internal) over a 10 year period, so $30M per year across all 26 institutions.
This came out to around $1.25M ACV per organization (10 year contract, 26 organizations).
In reality, some of the larger colleges (eg. CCNY) will have spent much more than the smaller ones (eg. Bronx CC).
In addition, each of the large CUNY colleges had budgets in the $150M range in the early 2010s [1] along with a separate $2B budget for CUNY's central organization (as mentioned above)
What seems to have happened is the above page treated overall TCV as part of a single budget, when in reality, each individual constituent org with it's own budget spent out of their own pocket for ERP, HRM, Headcounr (PS and internal), and Compute (on-prem)
This isn't egregious for the size of organizations that each of the constituent colleges and CCs in CUNY are.
The big question is, if you used the PSC's (imo flawed) methodology, what would it give for CSUs, UCs, and SUNY (all of whom have a similar structure and student size). I'd expect a similar amount.
On top of that, digging into their manifesto in the page above, I found this startling and horrifying stance the PSC has
> CUNYFirst will be part of the arsenal by which CUNY Central shoves Pathways down our throat
Basically, the CUNY PSC is opposed to a unified core curriculum, allowing students at all CUNY institutions to transfer between each other.
This is something that is diametrically opposed to student well-being.
Basically, the PSC at the time opposed NYC community college students from taking courses at CUNY's 4 year universities and vice versa, and was strongly opposed to interoperability and transfer of credits.
California's equivalent system (ASSIST) has had a massive benefit in allowing Californian community college students to transfer to CSUs, UCs, and private schools like USC and Stanford.
CUNY's faculty union (PSC) on the other hand is completely opposed to a similar system being adopted.
The whole point CUNY PSC was opposed to the CUNYFirst system was basically because the PSC was completely opposed to allows simplified inter-college transfer.
All things are possible. Here’s a scandal where the NYC government spent $700M to implement a time card system. CUNY has access to public authorities like the research foundation, which may fund stuff in addition to their appropriations.
> Organizations typically add 3-5x additional to either "pad" their request (in the event it doesn't get 'fully funded') and/or this is an opportunity for the university to higher for a bunch of roles they wouldn't have been able to get funded in the first place - so lots of things get buried in these numbers.
A factor of 3-5x seems insanely overboard. Thats arguably fraudulent. Ditto for claiming the money is for one thing and then using it for something entirely different.
I get that it’s a communist era style scraping to get by / work around a broken budgetary system life in academia, but that’s misappropriation of tax and tuition dollars.
Yeah seems like 50-100 % administrative overhead for each employee, hard to believe that someone would accept that, you could probably pay each employee a personal valet for that.
I'm reminded of the Oregon state health care exchange fiasco where Oracle took like $250M and never delivered a working product. Ended up in a lawsuit and a settlement.
Source 1: Only mentions anonymous sources. And while I would put partial weight on that coming from an actual journalist, this is the same professor who wrote the original blogpost.
Source 2: Same publication as Source 1, though I don't see a byline. No named sources.
Source 3: No named sources. The "About Us" features Lorem Ipsum and four photos of the same woman.
Source 4: A reddit post, no sources, and the amount quoted is not $600 million, but $250 million.
Source 5: A petition. References an Oracle press release that didn't show up in your Google search (most likely because it doesn't exist, or it would be widely cited).
Having worked for NY gov agencies for 7+ years, it's not that hard to believe really. Vendors overcharge crazy amounts all the time and get away with it without the gov client batting an eye. Experienced it first hand where I was on several projects where I did, without exaggeration, 95% of the work and my employer at the time charged $500,000 for one project and for another $750,000 in a single fiscal year.
Sure, it wasn't $600 million (seems like that $600 million to Oracle had to of been spread out too), but the work was simply formatting their website content from an old template in their busted CMS (pretty sure it was some ancient HP CMS) in HTML and had to write some minor CSS and some minor JS. I ended up automating it locally and then the rest of the time was spent copying and pasting through this convoluted process due to having to connect to their outdated locked down remote desktop with a super slow connection. Additionally, I was only being paid under 6 figures at the time and it took much less time than they were quoted for, though my employer sat on their hands to make it seem like it took that much time.
But yeah what the agencies were charged was unjustified irl... but on paper it was, they didn't push back on it or anything, and the cost of completing the projects was wayyy the hell less than what my employer at the time quoted them for. My assumption is they dont push back due to budgets not rolling over and wanting to maintain/increase their dept's annual budget. So it's not surprising they'd throw away that amount over the course of some years.
When you see how badly most academics and academic administrators are at actually running things from a business standpoint it no surprise the academic sector is in such a mess at the moment. Sadly this whole mess is funded by debt handed out that’s not dischargeable in bankruptcy for degrees of highly questionable value. It’s really sad when you follow the money and think about who and how stuff like this is actually paid for.
The whole thing is kept alive by the student loan program. Modify that or take it away and academia in the US would implode.
When you see how badly most business administrators are at actually running things from a business standpoint it no surprise the business sector is in such a mess at the moment.
If you've ever worked in a large corporation or government, you know that this is the norm. It's amazing that anything, anywhere, ever gets done at all in organizations larger than a few hundred people.
We as humans are very bad at this.
Companies like Oracle, Deloitte, McKinsey, etc. are experts at extracting large sums of money from large dysfunctional organizations.
I've worked in academia. I've worked in small, medium and large private companies and fortune 50 companies, and I've worked for small and large govt.
I have yet to see anything bigger than a small company run efficiently, and that's by necessity more than anything. The largest wastes of money I've seen were fortune 50/large enterprise companies yet people constantly point to gov and academia as the most wasteful.
My current job is a cycle of hype waves the executives and ceo buy into and oversell to customers in impossible timelines, the inevitable smack of reality when neither CEO nor customers are happy with the over-promised and under-delivered results, and the move to the next wave to capture more customers to replace the ones attriting because the result doesn't match the requirements despite waht they were told.
The actual users of our product seem to split fairly evenly on love or hate our products with good reasons while their executives and spending managers are almost comically willing to fork over more money over for little to no real benefits. There is very much "No one was fired for buying IBM" mentality in the corporate world, this seems like another example.
> Companies like Oracle, Deloitte, McKinsey, etc. are experts at extracting large sums of money from large dysfunctional organizations.
And IBM and Accenture, and etc etc
Feels like this entire sector *should be* ripe for disruption by a more skilled and far leaner organization for far cheaper, but my impression is that the majority of these contracts are some "business/partner" relationship things, and those that opt for these contracts are performing CYA by opting for these companies.
I've worked in a large government organization (the IRS) and at a large tech company.
I think it is fun to pretend like the issues in private industry vs public sector are similar in magnitude, but in reality it's not even close. I have never seen even remotely the level of dysfunction I saw in the public sphere in any private company.
Based on your comment, you've worked in 1 public and 1 private organization. Not sure this is the basis for coming to such sweeping and drastic conclusions.
We're all working from limited examples - this entire article is about only one. I know many other government workers in different branches who felt similarly.
Lot of people don't care about spending someone else's money efficiently. And in some cases spending more will make them look better and lead to better opportunities. And this happens on every level, lot of employees just don't care. And those that might probably focus on wrong things.
And no software people are no better, going for expensive tools, expensive cloud spending or just next shiny thing is often argued with some time to market or future scaling excuse. Or just wanting to do something different.
Why does no one on the inside even seem to acknoledge is publicly (I've seen enough internal discussions in the private sector to know they do complain on the inside)? Let alone try to fix the dysfunction? Does someone high up have perverse incentive or are simply too conservative to risk chipping away at the problem?
Ineffective corporation will eventually be driven under the ground by various forces, protected businesses have 0 pressure for quality and effectiveness, so unsurprisingly there is little to none.
Anybody who dealt with good amounts of state bureaucrats can see this rule in plain sight. Bonus points if you actually know some privately. Then they have the balls to want same lifestyle as somebody working hard on themselves and their careers, doing sacrifices they wouldn't even dream of accepting. Of course success never materializes, that evil capitalist world is against them.
I recently left a software dev job in academia, and the amount of inefficiency in the organization is insane.
As part of my off-boarding, the lead engineer mentioned she would ideally like to hire 5 more developers to the team. That would bring the team size up to 15 (8 devs, 2 devops, 2 Ux, 1 graphic designer, 1 pm, 1 eng manager). The team maintains two things:
- The static website for the library
- A pretty basic image server and viewer for the library and museum collections
Sure, the library needs a website, but that shouldn't require more than a person or two to maintain. The image viewer was only used by a handful of people.
But it doesn't matter. The team gets funded because the students will keep paying their tuition. The engineers can keep sitting around watching youtube all day and the world keeps turning.
Perhaps most egregious example was in my first 1:1 -- my manager said, "Don't expect much output from [SENIOR ENGINEER X]. He isn't a good engineer." The organization isn't willing to fire anyone. As a result, the people in charge are the ones who have been around the longest.
That said, it's risky to fire anybody because hiring is so difficult. The salary bands are set at the university level for all employees, which means the maximum a software engineer can make is much lower than market rate. To make matters worse, working "in person" is mandated by the library dean, and the university is in a college town in the middle of nowhere. The interview process also includes NO coding sessions although I'm not sure whether that's due to some corporate process or just incompetence.
To be fair, these sorts of issues aren't exclusive to academia, and I've seen similar issues in large organizations. Somewhat paradoxically, the more bullet-proof the business-model is, the more potential there is for rot to grow in a company.
> Perhaps most egregious example was in my first 1:1 -- my manager said, "Don't expect much output from [SENIOR ENGINEER X]. He isn't a good engineer." The organization isn't willing to fire anyone. As a result, the people in charge are the ones who have been around the longest.
Kidding aside, where can I find one of these jobs? I'm burnt the hell out and would love to lick my proverbial wounds for a year or so doing this.
I'd love to get a job like this seems amazing. Imagine all the gold plating you'd get to do. All the experiments and optimizations. You'd have the best library website ever.
When you're on a team with 15 people, that means a lot of process to push through to get anything done. One of the other engineers was the owner for analytics, and as a result we had basically no analytics. Furthermore, the team lead would often say things like "developers aren't designers and shouldn't be thinking about the product". I wanted to implement a basic semantic search to help people find the resources they were looking for (#1 pain point from talking to people), but that was shut down at every turn.
There actually was talk of firing someone once, but the reason was he didn't always arrive at the office at 9am sharp, and wasn't in any way related to his output.
This doesn't sound like the academics', or even administrators' fault- this is being imposed by the state university system. Reading between the lines the change is a response to political pressure to reduce costs and impose restrictions on curricula.
>> Reading between the lines the change is a response to political pressure to reduce costs and impose restrictions on curricula.
Reduce costs by spending huge amounts of money to lose capability? This smells like someone got an "incentive" to spend public (govt) money on some corporate project. Not sure why anyone wants to impose restrictions on curricula, but that'd be a kind of separate thing.
I'm an academic at a large state university (not SUNY). Faculty, staff, and students generally have very little direct say in IT-related matters. These decisions often come down from central admin, through a process that is just as mysterious (and sometimes infuriating) to us.
I will also chime in. Having contracted with multiple large, state universities, this is the norm. Staff and faculty have little, if any, input into the systems the university uses, and are often just as confused as the rest of us.
Every institution I've worked for had a check-off for IT and central admin if software purchases were requested. These are well-known to be poison to most initiatives without a Dean level or above pushing for it.
I don't think people that criticize non-admin university employees have any idea how these things work. Not only do they not talk to the people that do the work to find out what they need, they're not open to feedback on the garbage they've cobbled together after they put it in production, and every decision is made assuming faculty, staff, and students are always wrong and they're always right. I could write a book about the things I've seen.
The university I was at had this amazing process for internal billing. Everything cost heaps and required entry in an arcane system. If you could do something inhouse, you were supposed to, either though the cost was astronomical. A $100k bit of software from an extra vendor was definitely $200k+ once the various IT departments (there were at least 4) had done their bit. There were managers all the way down and all were very important.
I have found that most academics have zero interest in how the sausage is made. In fact, I would say there is a willful ignorance to understanding the intricacies and complexities of what it takes to run a University system and the precarious financial realities of most colleges and universities.
However, every once in a while an academic decides to take on administrative responsibilities to fix all of the things they perceive as broken or to show everyone how smart / right they are. Usually the first year for them is incredibly difficult for them and everyone around them and they make a true and terrible mess of it. At this point, after a full year, they usually know just enough to realize how little they know about running a college, managing people, or generally being a leader. They then react in one of three ways:
1. Resign from their admin job and go back to just teaching as if nothing happened
2. Become humble and begin actually collaborating with people and not blaming everyone for every thing that isn’t as they see it.
3. Double down and truly blow everything up until they are either fired or whatever it is they are running collapses in on itself.
This isn’t everyone. The faculty who transition best to leadership tend to be pretty humble to begin with.
A bunch of years ago, at college, I built a class management platform for my university. It won me $1k (which was an insane amount of money to college me), and I met with president of the univeristy to pitch him on using it. They were potentially buying Oracle software, and I found myself up against them. At the time, I felt like this professor.
They obviously went with Oracle. I'm sure they spent a ton of money, and I'm sure it was the right choice. I would have gotten bored of it pretty quick. You don't pay Oracle because it's a good deal or a good product... you pay Oracle so you never have to think about it ever again.
I don't really have a point here. I wish there were better options in the market. But I certainly don't want to build them, since it's a boring problem with bad customers (edtech is a horrible sector to sell to). Oracle has a price point that makes it worth it to them, and they have customers willing to pay it.
Hopefully someone sees this blog, and rather than be annoyed at academic/government waste, sees a big market they can dominate with a better product. But given how it was written in 2013, I'm not so sure.
They're also paying $5M per year to maintain the servers and software. In theory they don't have to think about it but that just highlights the waste even more. I wish I could spend $5M without thinking.
> so in perspective it's a very small amount of money
$5M/year could pay for at least another 10-20 professors which is like an entirely new department. Let's also not forget the initial $600M spent which (assuming they'll use this software for the next 30 years) will be $20M/year bringing the total to $100/student/year which isn't negligible.
More broadly speaking, I have a gripe with people bringing up percentages of budgets when discussing how much value something brings. My college had a startup fair where students pitched some super innovative ideas. The grand prize was $5000. If we had $5M, we could give that grand prize to 1000 teams and imagine how much more valuable that would be for the school compared to another feature bloated, overpriced piece of software.
> edtech is a horrible sector to sell to). Oracle has a price point that makes it worth it to them, and they have customers willing to pay it.
The way businesses deal with this is by selling more expensive software! That's why Oracle is getting rich by selling crappy software to difficult sectors that nobody wants to touch.
Well sure if you can manifest money like a daydream, then you won't have to think about it anymore. I don't know though, maybe there are some folks out there who do in fact spend time thinking about unnecessary costs?
My understanding is that this network of universities is in fact a bunch of individuals who sometimes care about costs they are generating for their customers.
This is a post by Prof. David Arnow on the blog of the Brooklyn College’s professors’ union about CUNYfirst, a PeopleSoft-based course registration and HR system sold by Oracle. Posting because the system recently got some attention on Twitter: https://x.com/ChocolateyCrepe/status/1836171439965446441
It seems like there should be 5-6 vendors of 'University HR Software'
You want 1,000 licenses for it? That will be $5,000 a year, for a total of $5 million
It's going to us a year to implement it, we're going to send out 25 people to get it up and running, train your users, etc. That's another $25 million.
We'll spend the next year building integrations to all of your other software and systems, that's going to be another $25 million.
These quotes will all vary +/- 25% depending on the vendor. Schedule a 200 hours of meetings, trainings, etc for 500 people involved with the new software. That's another $5 million.
Well CS departments are rarely staffed by people who are interested in building actual software (if they were, they could just work in industry making significantly more money), and HR software is probably the worst vertical for a bunch of kids with no experience working in.
My school tried this in the early 90s: they held a contest to make class scheduling software. I don't think it worked out- I certainly did not enter it.
This was to replace the previous system: professor John van Alstyne just did it over a weekend.
I worked in IT at a CUNY school at the time. Just hilariously poor and unintuitive. Every student was given an Employee ID number. Course registration was basically handled by an e-commerce addon.
If you think about, $600M is enough money that someone somewhere could create an entire new company and staff it with some of the best developers just to get this contract...
Regarding the other bidders dropping out, what exactly was the primary expense that gave it an estimated $1B total cost? $600M can build out a greenfield software platform, so there must be more to it.
Garbage overpriced systems that everyone hates, and which substantially hurt the organization... How do you land those sales? (Joke: "Asking for a friend.")
Some bad ways that big-ticket purchasing decisions are made:
* Committee of people who don't know what they're doing, and/or who can't coordinate to arrive at a good holistic decision/solution.
* Person who wants this done for good reasons, but doesn't know what they're doing.
* Person who wants this as an accomplishment credited to them, but doesn't know what they're doing.
* Person who is mainly thinking "nobody ever got fired for buying [old big-name vendor]", and everything else is secondary.
* Person who is bribed by vendor (e.g., immediate cash, quasi dates with attractive salesperson, career rotating door with vendor).
Other ways?
(I haven't directly seen the bribery way, though heard of it in news stories. I've definitely seen all the other bad ways happen.)
1. Be willing to spend 5 and 6 figures (unreimbursed) responding to lengthy RFPs.
2. Work with an established Systems Integrator (who will take a large percentage.)
In government procurement positions, I would be in favor of giving a small %-ge (to the tune of a few hundred thousands of dollars) of money saved from budget to the people in charge of procurement.
The California State University system contracted with PeopleSoft to consolidate their various non integrated systems back in 2003. Initial costs were supposed to be 350 million, but instead ballooned to over 700 million. We still don't use the full functionality of the suite, even though it's all been payed for.
And it's the absolutely worst system ever, a complete pain in the ass in every way. If you've never touched PeopleSoft, they not only spent hundreds of millions on a system, but on a system that sucks away years from your life-force.
Why not let the computer science students take a crack at building some of those systems? Maybe not the an HR system, but why not a course registration system? Maybe not for the whole university, but maybe for just the computer science department?
The university would get work for free and the students would get real-world practice with building production code.
I get that there would be risk, but if it was under the supervision of professors (who hopefully are good at building, not just lecturing theory), I think there is an opportunity there.
> I get that there would be risk, but if it was under the supervision of professors (who hopefully are good at building, not just lecturing theory) […]
Good joke here! If you are actually serious, please tell us which university you encountered where the majority of professors actually did something productive in computer science
Sorry if that came around as rude. From personal experience I have not met any professor in my life who could lead such a project, let alone with students that don’t have any work experience… And you imply there should be multiple of them for the project to succeed
Someone has to maintain the code. Can you imagine making students doing that kind of work? They'd rightly asked to be paid for it. And getting faculty to manage the students doing that kind of work, they'd definitely want extra pay.
>Oracle-PeopleSoft did not drop out. However they warned CUNY that for that level of funding, they could not, would not CUSTOMIZE: they would only CONFIGURE
In this case, I don't think Oracle is to blame.
They explicitly warned about the 'limitation,' yet the project proceeded nonetheless.
In my opinion, for this project to succeed, it should have been built from scratch. With the $800 million invested, they could have assembled a team of seasoned and junior developers to get it done. Just my two cents.
I find it really hard to justify spending $600M on a system like that? Look, if you had 500 skilled FTEs working on the project for 2 years, at $250K per FTE, that would be a total cost of $250M. Say 50 FTEs for ongoing support at $12.5M/year or $125M for 10 years. So $375M.
But the above numbers are hugely generous. This is not building an ERP from scratch. Do you really think it would take 500 people (say 400 engineers and 100 non-engineers), to build and deploy such a system? I would imagine you could get it done (and done right) with half that many, or less.
IF you go to a body shop to get a dent pulled, you will receive one quote. If you tell them insurance is paying you will receive a second, much higher, quote.
The dealer will say it's because the insurance company is such a hassle to deal with that they have to dedicate employee time to jumping through hoops to get paid. The insurance company will say the quote is higher because the dealer sees the insurance company as a big bag of money and if the insurance company doesn't ride them then prices of insurance will go up.
Your analogy is somewhat backwards. Insurance companies squeeze body shops pretty well. If you go in there and ask for the same work you will pay more.
The real difference is that often times a person will just say make it look okay and I'll be fine. Whereas insurance work is basically making a car good as new because that is legally required. If you try to get the same level of service at retail you will pay more.
Interesting. I've been lucky in that I haven't had non-cosmetic damage repaired, but it always seemed like there were several iterations when insurance was involved.
With Oracle? 400 Engineers is likely way to high, it's more like 100 Engineers and 900 non-engineers. Oracle likes to throw in a ton of people into contracts with lie that they will provide "white glove" teaching and various other things.
The trick is the find some way that a vendor can monetize it. Sometimes you have management consulting groups building crappy unmaintained (except by them, if you extend the contract) OSS.
One thing that seems to work, sometimes, is having cloud vendor support. Could you support maintenance of an OSS platform that AWS (and/or their competitors) operates - that way AWS could "win" the contract, but the OSS system gets funded with a core team and various groups still contribute to make it better
I agree. Any tax payer money used for creating software should be required to be open source. Ideally, governments and similar organizations would utilize open source software, then use contractors to modify, support, and maintain it (again, releasing any changes as open source).
Contractors would then be responsible for providing excellent support, not some huge bloated product.
Pretending it is impossible to get a semi-democratic government to work for the people instead of just very difficult is defeatism. In fact, getting people to believe "It's hopeless" is a big part of the way bad governments maintain control of people, like in Russia or in the USA
I'm not saying it is impossible, I'm only talking about the situation in the US. Currently it is dominated by the billionaire tech and financial oligarchs and we need to do something to change it. It doesn't make any good to sugarcoat this reality.
And yet 90% of all startups fail, which could also be seen as a indicating a certain level of incompetence in the private sector.
Consider perhaps that incompetence is distributed across the economy, and government and private industry share in that, with successful and unsuccessful projects as a consequence.
Drexel University nearly bankrupted itself in the 80s, among other things, by hiring Lockheed (then Martin Marietta) for a similar boondoggle project. The project was an unmitigated disaster that never delivered anything, and the university had to pay to exit the contract.
My goodness, the world never changes. Saw the same "new enterprise software much worse than the incumbent, terrible UI/UX, everyone hates it" dynamic play out *30 years ago* with SAP R/3. When I check back in 2054, expect it will not be any different. Technology changes, but people and organizational dynamics largely do not.
As a somewhat related PeopleSoft story, the government of Canada paid IBM and a variety of contractors $3.5B over several years for a PeopleSoft-based payroll system called Phoenix.
The system was a disaster. It never worked properly. So the government is throwing hundreds of millions towards its replacement.
to manage means to extract-value-from and $600M for software is a lot of value extracted.
I'd bet huge that there is a layer of managers who don't see themselves as being accountable for domain competence in anything they manage as that's what the consultants are for. Consultants mean headcount and budget to manage- which is the definition of success in an institution. they run an organiation, the mission is little people IC problems.
It's not broken, it just works for people you can't see.
>> I'd bet huge that there is a layer of managers who don't see themselves as being accountable for domain competence in anything they manage as that's what the consultants are for.
Ah the old "management is a generic function" argument. A CEO from Coke is qualified to run GM or Intel. Just have the executive team give me all the information and spell out what results in the biggest $$$ and I'll "make the decision". I thought that silly notion was sunset some time ago...
it's how institutions work and it's very alive anywhere with inclusion programs of any kind. to hire managers based to anything other than strict hierarchical domain competence depends on the premise that managers are generic low agency minders of things that are already creating value themselves, and a transferrable/interchangeable function. nothing outdated about that at all.
Does anyone know what "pathways" is and why they wanted so much central control to set that up? It seems that there was some sort of power angle to all this.
> There has been no entity in human history with less complexity or nuance to it than Oracle. [...] This company is about one man, his ego, and what he wants to inflict on humanity. That's it.
With that money, Larry begged Jensen to take it and deliver him the GPUs. Onto the new grift. In yesteryears it was database and now it is AI. Benedict Evans always compared AI to databases. Oracle of AI is THE Oracle of Databases.
I have been negative on Oracle since the ‘90s, when I had to work with an installer CD for Oracle software —- an official Oracle CD —- that:
1. Had multiple installer applications on it, with no indication which was “the” installer application.
2. On opening the installer, asked me to select the install file to act on, again with no clear direction.
3. Had “help” files on the disc, in HTML format, which contained broken links to other files on the disc.
At the same time my coworkers, experienced Oracle DBAs and developers, with full paid-for support from Oracle, spent an entire summer trying to install an Oracle development environment, and failed.
All of which to say, yeah — $600 million sounds about right, as long as it turns out the software was never successfully implemented.
This is why I never, ever, buy or recommend software without actually using it along with the people who’ll be dealing with it everyday. “Oh, devs will have to install it on their laptops? Ok! I’m a dev. Send me a test license and I’ll see what it’s like.”
I’ve made mistakes, sure, but I’m proud to say I’ve never signed off on anything I wouldn’t be personally willing to endure.
In brief, administrators are tunnel-visioned on checking boxes without being mindful of the fact that if you only need 1% of the features, the other 99% represents unnecessary complexity that will directly impact the usability and maintainability of the "solution".
> CUNY Central was so eager to have a centralized MIS tool to use for its own centralizing, corporatizing agenda, that it totally ignored the implications of the Oracle "configure-only" limitation: business processes would have to be made to fit Oracle, not vice versa. Capabilities that we now have will vanish. The staff, the faculty, the students would just have to "adjust" (the technical term being "suck it up").
This is an interesting perspective. From what I've seen / heard from others, it's generally better to adapt processes to the off-the-shelf tool than to try and customize or build from scratch to accommodate your bespoke processes (especially in the business operations realms). For one, the organization is likely less unique than you think, and those bespoke processes are as often a function of some early employee's preference as it is a genuinely good reason. For another, customizing software is not just a one time cost, since every subsequent update / upgrade is likely to require additional work (or at least testing). And finally, in most cases the closer you are to a vanilla, standard process, the more likely you are to stay in compliance with local laws and regulations.
Though I suppose it's possible that the imbalance is just due to the fact that it is much harder to quantify the costs of using a suboptimal (for you) process than it is to look at the procurement contract for a custom solution.
I've had the opposite experience. Any attempt to adapt processes to an off-the-shell tool have always been a subpar experience for everyone. I've brought a few of these in house as fully custom software and the end result has been a better user experience and faster changes. If we can buy something that fits the need, we will. But if we can't, we build. And we build a fair bit.
I disagree with "the organization is likely less unique than you think". If you're big enough you will have unique requirements that nobody else has. I'm in the middle of that now in a project to install some industry standard software that runs the whole business and we have to customize and add custom integration into it. I wouldn't want to build software this complex in house but if I was given the resources and tasked to do it, I could, and it would be better.
There are two different scenarios.
Scenario 1: You're doing something that every other business is doing. E.g. ERP/accounting, sales, contact center, etc.
Scenario 2: You're doing something few other businesses are doing. E.g. your actual customer business, creative, etc.
(1) is amenable to making your process fit software, to good results. (2) is usually a train wreck.
Unfortunately, figuring out if your thing is scenario 1 or 2 is non-trivial.
Canonical example: EMR/EHR systems in healthcare. You think they'd be the same... but actually there are so many integrations with other systems and/or different sorts of specialists, that a real world implementation has substantial functionality gaps (papered over with custom work).
My impression is that most people don't understand just how awful most commercial business software actually is.
One thing our business does that every other business does is vacation and overtime tracking. We have a custom in house application for that and we've yet to find a commercial replacement that is, in anyway, half decent. For most Payroll/HR systems, this is merely an add-on feature and doesn't get much attention.
For overtime, integration with our financial system allows overtime to be charged to the correct files and this is something that nobody does (or does well). Probably doing just this little bit makes this project pay for itself.
Commercial business software manufacturers are isolated from the users and in a way isolated from consequences as long as they fulfil the contractual obligations (which practically never has a 'make the users happy' stipulation).
I think this is why you only see innovation via alternatives vs within a product.
E.g. Salesforce, Workday rising up to replace incumbents, but then themselves becoming stale.
Beside the lack of attention, you also have gargantuan legal requirements you need to integrate. Which change all the time. Sometimes a few per country.
This is for everything: pay, vacation, etc.
It is really complicated.
One advantage of building in house is that you're only building for your own company. This is significantly less work than building commercial software for multiple clients (which I have also done). I can't overstate how much less work this is and how much of a better experience it can be for users.
As an example, for calculating annual vacation entitlement, we have some pretty complicated rules. But every company in every country has their own set of rules so most HR software doesn't bother calculating it -- you just figure it out manually and input it every year for every employee. But because we just have one "client" our rules are just code that we can change as needed and can arbitrarily use whatever information we have. This saves HR a ton of manual work all the time. But this only works because it only needs to be one set of hard-coded rules.
> has their own set of rules so most HR software doesn't bother calculating it -- you just figure it out manually and input it every year for every employee
Beg to disagree. This is the complexity that large ERP firms handle and why Oracle, SalesForce, etc are expensive to implement. They figure out the commonality (if any) and build for it. Then they add on features specific to countries they target and then they add the ability to configure for your own situation (to a certain level).
PeopleSoft did this for Payroll and workforce administration which is part of how they cornered the market for HCM.
The problem when building it yourself is that this is usually done by "generic" developers who discover edge cases (often together with the requester) that threaten the whole model.
A company doing payroll for us ("us" being a multinational company) asked for a "typical payroll" to start with. Fortunately we had experienced people on the pay side who discarded the company because of this question (for one they should know, and for two they should know that "typical" will cover maybe 60% of the cases -- I thought that this was an exaggeration until I discovered the reality of calculating pay in France)
A good company specializing in "pay" or "vacation" (which are very regulated over here) will know the "typical" case and the edge cases.
I think most moves to EMRs were insufficiently disruptive, e.g., electronic orders recapitulate the old paper orders without using the opportunity to reduce ambiguity and insert constraints to prevent common errors.
The problem is that the paper form is the interface, because the ecosystem was designed around it.
10 years ago or so, I asked a health insurance company why a specific digital form could only list up to 16 diagnostic codes (otherwise a duplicate with the additional needed to be created).
They thought about the question for a second, then said that's how many were on the paper form the digital system had been created from (35+ years ago).
> If you're big enough you will have unique requirements that nobody else has.
Definitely agree, and that is very interesting to hear. I only mean to speak to my experience, and what I saw in a lot of cases was unique requirements that were due to the organizational equivalent of tech debt (ie things like "our books are organized in this unique way because we acquired this other company a many years back but kept their stuff separate because it was the path of least resistance at the time").
That’s a legit issue. What’s the value of changing a deep seated process?
I have definitely seen examples on both sides of that question. Especially in a place like a public university with multiple collective bargaining agreements. The unions aren’t going to accept significant change without some sort of cost.
Typically, since processes are built around the system, nobody understands the actual business needs.
Nearly all our long-term tech debt come from having to work around limitations in the commercial software we use.
> If you're big enough you will have unique requirements that nobody else has.
the problem is when EVERY process is that way.
yes, each business has unique aspect to their processes, but when every process is heavily customized by people who have no business designing processes & applications, organizations start to hamstring their flexibility and scalability.
Its like hand-making a Ferrari with completely custom parts when what you need is a Toyota Camry. If you aren't gonna race with it, its a waste of money.
I have been working in the ERP space for over a decade. and almost ALL customers have no criteria for when to customize or keep a customization. They don't do any cost benefit analysis or any strategic planning beyond building a customized tool for the thing immediately in front of their nose.
Personally I hate software customizations. I prefer to build from scratch using proper software development tools than use whatever piss-poor customization system commercial software typically provides.
I recently sold my director on building some software from scratch rather than try to combine two commercial products together for some Frankenstiened solution. The motivation for Frankenstiening it is that we are paying for both services so we should use them (and at least one is very customizable). Only after 6 months of failing to get these systems to play together in an acceptable way, I finally asked a team member to spend 3 days building a mock up of a new app. From there it was quickly approved and we hope to roll out next month. I only wish I had done that sooner; we would have had way more time for development.
Exactly this! I recall a large survey of SAP deployment projects in the late 1990s. By far the most successful consultancy, out of Chicago I think it was, had it written in their contracts "you'll change your business processes to match SAP; not SAP to match your existing business processes" (more or less). By turning away clients who could not accept that, they had happy clients, happy employees (little burn out), and no runaway costly never-ending death march projects.
There's a bit of selection bias going on there though. The reality is that SAP and similar products are designed for a business that works a certain way, and so obviously businesses that fit that profile are most likely to get value from using the tool. However, there's a reason other businesses don't work that way, and often retooling to work the SAP would be a net negative. Sometimes retooling SAP to fit the business is also a net negative, in which case the right choice is just to not use SAP, but I've certainly observed cases where there was a benefit from refactoring the tool to fit the business.
On "retooling SAP": SAP deliver their systems with all source code and dev platform included, and that may help convince some customers to go for SAP.
However, those that embark on deviating from the well-trodden path are going to be in trouble soon: after every update, potential changes made need to be re-done or edited or at least tested. So as the parent suggests it's really better to adjust the business process if you can.
> So as the parent suggests it's really better to adjust the business process if you can.
That's another way of saying that there are serious trade-offs going that way that need to be justified... which may also be true for the path of adjusting the business process.
There's a ton of variety out there in the real world such that you'll always find a few businesses that match almost any scenario you can imagine. So, yes, for some business doing a process a certain way is a competitive differentiator or advantage ... or even a necessity for their particular industry. For these businesses banging the SAP square peg into their special BP round hole is worth attempting. Even better might be just building their own custom round peg. I'm suggesting that many (not all) businesses are doing a BP a certain way for no compelling reason whatsoever other than that's the way they (almost randomly) picked to do it decades ago and could easily change to a standard practice without loss of competitiveness (maybe even gain by focusing on what does).
Exactly what I'm trying to say, though written better.
> There's a bit of selection bias going on there though. The reality is that SAP and similar products are designed for a business that works a certain way
ERP products are designed following "standard" or "best" practices/processes. It's common to see companies first contract a consulting company to "re-design" their processes before they then try to implement an ERP system.
s/ERP/any other business software product/
...and yet there are all kinds of segments where customized tooling is more the norm than otherwise. It just depends on whether the deviation from the norm is a competitive disadvantage or a competitive advantage. There are a lot businesses where in at least one case, it is an advantage.
That's just days it's visible to be selective in your customers, not that all businesses have the same processes or would be better off not customizing.
> From what I've seen / heard from others, it's generally better to adapt processes to the off-the-shelf tool
I have never seen this work even once. I've actually built entire businesses on the concept that people are impossible to change, but software is easy to change.
Not ERPs.
Customizing ERPs is where consulting firms make money but the ERP vendors themselves advise against this because it becomes expensive maintaining the customizations as new versions of the software and more features are released.
Did you use much off the shelf software? I mean, I'm guessing you used databases and frameworks, but were the business process and UI components off the shelf or custom?
Because it's a tradeoff.
I use as much off-the-shelf as possible. But there is real work involved in adapting software to a company's culture.
If you do it right, 99% of the work has already been done for you by an existing SaaS or FOSS project, and "all" you have to do is customize it. And generally this doesn't mean "change the config" but rather assemble different building blocks into the right shape.
This can still be a multi-year project, even when you're writing almost no code. It takes a long time to understand an existing company's culture, processes, and needs.
Makes sense, this is just my personal experience so it's always interesting to hear what others have seen. I recently learned about Mechanical Orchard[1], which seems to have the same thesis (better to update legacy custom software with a modern custom solution rather than migrate to a modern off-the-shelf solution).
[1] https://www.mechanical-orchard.com/
My employer used Peoplesoft, in the days before Oracle bought it. Of course, you can't set up Peoplesoft on your own, you bring in consultants. When I reviewed my notes from all the pitches, I found that they all had said that we should modify our processes to suit Peoplesoft. I guess we did--I was new there.
I would also say that painful, unduly costly implementations seem pretty standard for ERP products. I have heard such stories about SAP.
I'm not surprised at that. It ties in to my responses [1] [2]
1. https://news.ycombinator.com/item?id=41584410
2. https://news.ycombinator.com/item?id=41584391
best example: the €500m disaster Lidl had with SAP [1] https://www.computerweekly.com/news/252446965/Lidl-dumps-500...
The problem is, the business process shipping with Oracle may be grossly inefficient. Or worse, the former Siebel module and its integration with a former PeopleSoft module may be replaced by some new thing with yet another business process.
In any case, CUNY would likely save $300M hiring clerks armed with excel and paper forms.
Seems like a lot of money wasted, mostly to advance a “lowercase p” political dispute to consolidate HR functions with a questionable benefit.
One more reason to adapt your process to the tool is that it can make hiring/training easier.
Now you have two problems: rolling out the new system and changing the business process. And you'll solve them simultaneously, and if anything goes wrong, you'll roll both back...
Business processes change all the time. Often you are better off changing - if you are more like everyone else you can hire someone who knows your processes and they are useful without a lot of training. So short term costs, but long term better.
"IT will tell you what the new process has to be" is usually a pretty tough sell.
I work on bespoke ERP software, it's not an undertaking to be taken lightly, but you are missing a key point.
Every process that doesn't your software doesn't accommodate becomes a manual process. There is an employee in a sibling department, using other software, who's full time job doing the work of one of my applications in excel.
I know hating on Oracle is en vogue, but I struggle to believe this $600M figure.
a. I use to work in this space, even a $6M deal would be massive (let alone something 100x bigger).
b. The ENTIRE cuny budget in 2013 was only $2B [0]. This isn't their IT budget, this is literally the entire budget to run the entire university system across multiple campus, faculty, buildings, etc.
c. Because Higher Ed is known to be so cheap, especially in the late 00s - big tech companies charged accordingly (which was at a massive discount relative to most other accounts).
d. even if this $600M figure was an aggregated figure over multiple years, staffing and auxiliary costs - it still wouldn't come close to this figure.
e. an expenditure this large would definitely be called out in CUNY annual financial reports, and I can't seem to find any reference to it.
[0] https://www.cuny.edu/wp-content/uploads/sites/4/media-assets...
---
EDIT: I did find reference last year to a $175M funding request to migrate from on-prem PeopleSoft (Oracle) to cloud.
Though, what I've historically seen is that only 10-20% of the funding request actually go to the software vendor. Organizations typically add 3-5x additional to either "pad" their request (in the event it doesn't get 'fully funded') and/or this is an opportunity for the university to higher for a bunch of roles they wouldn't have been able to get funded in the first place - so lots of things get buried in these numbers.
Lastly, the figures are also typically multi-year. Like 5-years being asked up front for approval.
Said differently, it wouln't surprise me if the true annual migration cost from onprem to cloud PeopleSoft might only be $10-20M
https://www.cuny.edu/wp-content/uploads/sites/4/page-assets/...
I cannot imagine any college paying 600 million for such a thing. I suspect the writer is very much mistaken on the price and a bit paranoid on the College's motives. Perhaps it is 600 million over 10+ years.
Why they would ever go with Oracle is another mystery as there are a number of vendors who specialize in this type of software for colleges (some of it good, most of it bad), so it would be foolish to explore a solution from Oracle that needs such a high level of customization to fit the needs of a University.
These kinds of mysteries usually have a frustrating and mundane answer. The people with the power to make the decision:
- were incompetent and just went with a big name, or
- made the decision due to salespeople (social pressure and/or gifts), or
- had interests that did not align with the org (ex. resume padding)
But who knows, maybe there's a good reason they went with Oracle.
You are likely correct. My best guess is they hired someone in IT from outside of higher education that had extensive experience with Oracle and had deployed it successfully in the past and so it was always going to be Oracle even though it was a poor fit.
We don't know the true scope of this, but apparently $600m was low balling in this story:
>The negotiations that were the run-up to the purchase of CUNYFirst were a travesty. The project required an expenditure of up to a billion dollars to do it right. CUNY Central offered far less. All but one of the bidders dropped out as a result: the project could not be done properly with what CUNY offered. Oracle-PeopleSoft did not drop out. However they warned CUNY that for that level of funding, they could not, would not CUSTOMIZE: they would only CONFIGURE.
I do agree that it sounds like this was some 10+ year project with heaps of support. If this account it true, they went with Oracle because they were the cheapest (and were fine with the compromise of no configuration to hit that goal).
If that hunch is true, I can see some $600k-$1m/year average for support of an entire chain of universities (remember that CUNY is a systetem of 10 colleges and more CC's like a UC, not just one campus) being peanuts that no one wanted to touch.
Seems they got what they paid for:
>The actual cost far exceeds the $600 million dollars that go to Oracle. Because processes are now much more inefficient, more people have to be hired to do tasks that were formerly automated.
> - The interface is laughable: It looks like an early-90s update of 3270 bi-synch technology. Web 2.0? Ha. Not even Web 1.0.
> - Because CUNY wouldn't pay for customization, we had to renumber our courses. This is just one of many, less visible to faculty, changes that CUNYFirst has forced.
absolutely not surprising as someone who's used Peoplesoft. That stuff felt ancient even in 2003. to sign onto that 10 years afterwards for a very stagnant software is just insanity.
-----
now of course I'm taking this at face value. It could be exaggerated or from a biased source. But in my biased opinion, who in the last 15 years isn't biased against Oracle's software? they will probably literally die out as their clients age out of the job market.
My first thought was for $800 million you could found your own startup to write an HRMS and have enough left over for a fleet of super cars.
I once submitted an RFP response for what was literally a CRUD app for a government agency. Medium-ish (to me) traffic. Needed to be rock-solid. Needed to be developed to spec, hosted, and supported (including tech support) for 5 years. I bid a little over $2million (I personally would have made about $500k of that). After a bit, the RFP was withdrawn and given to a sole-source provider for $100million, with the excuse that it was so complicated (HN would laugh at this if you saw the spec) that only one company could do it. So, that company gets it. Outsources everything to contractors in Pakistan. Site is constantly glitchy. And this next part is not a lie (and was def. not part of the spec) -- it only worked during business hours (m-f, 8am-5pm)! However the users of the site had to do work every day due to legal deadline requirements. They just had to set on the data until the next workday morning. I suspect that the provider spent about $1million or less for the entire contract and just kept the rest. So that is how govt. contracts can go.
With government bureaucrats with no penalty for failure and no expertise in software development guiding the design? And “lowest bidder” contractors to implement whatever shitty design the bureaucrats came up with?
Hahaha recipe for failure.
Well, San Francisco tried that for $40m and all they ended up with is a system that doesn't work and an Indian contractor that they made rich.
Sounds like poorly aligned incentives.
SF's watchword is "MORE FUNDING". Money is like violence: if it isn't working, you aren't using enough.
That's kinda the only lever a lot of government has.
If you don't build out a bunch of civil positions then you can only solve problems by throwing money at ideally somebody who can.
That's my dream. One day I hope to recommend someone spend more money on me because I suck at what I do. So long as it's not their money, they should be quite willing.
If they paid WITCH company to realise it...well, they got what they paid for
I agree. The $600M number seems way too much.
>Though, what I've historically seen is that only 10-20% of the funding request actually go to the software vendor. Organizations typically add 3-5x additional to either "pad" their request (in the event it doesn't get 'fully funded') and/or this is an opportunity for the university to higher for a bunch of roles they wouldn't have been able to get funded in the first place - so lots of things get buried in these numbers.
As someone working on implementing an Oracle solution for a large bureaucratic MNC this is also true. The padding often is 3x-5x with 5 year run rate the funding asks can be crazy. For people who are just looking at numbers it can seem crazy. But for anyone who is implementing it or renewing contracts they know the real numbers.
The migration costs though might be higher than $10-20M. Contractor costs can be crazy too. Sometimes costing multiples of the software costs.
In my current practice I do notice that the vendors - both software and third party implementation partners - have a chummy relationship with a decision maker. This creates a lot of misaligned incentives when it comes to "company's" money.
This is absolutely right - the education space is absolutely stingy and will balk at far lesser amounts; so the $600M seems difficult to believe.
I've seen this happen before with big companies - they don't have $100k to pay for software which would save them $5M / year.
Why? They'd signed a contract with SAP who'd sucked up their entire budget.
Not quite true, especially not for the higher end research universities.
The University of Washington is projected to spend $2B on modernizing IT infrastructure (which does seem like a reasonable amount).
In that number though is $340M to update it's accounting system. Not quite directly comparable to $600M for HR, but not completely perpendicular either
It's $300M over 10 years split across 26 institutions.
In fact I'd say this is a better deal than the one the UW Systen got.
A lot more institutions, but in terms of total number of students, about 4x the UW system, ~2x amount of administrative staff, 4x the academic staff.
Not saying the UW got a good deal of course, it's pretty clear based on the progress of the modernization that UW didn't.
Plus one on this. I've almost never seen a 9 figure deal in my life, and the only ones I've seen are massive core infra level deals by mega-corps. I absolutely cannot imagine CUNY spending more than most F50s spend on HRMs.
Edit:
I did some digging into the Tender [0] and it looks like it was $300M total across all 25 universities and colleges in the CUNY system along with the CUNY administrative portion itself and it was a full bundle of HRM, ERP, On-Prem Compute Resources, and headcount (PS and internal) over a 10 year period, so $30M per year across all 26 institutions.
This came out to around $1.25M ACV per organization (10 year contract, 26 organizations).
In reality, some of the larger colleges (eg. CCNY) will have spent much more than the smaller ones (eg. Bronx CC).
In addition, each of the large CUNY colleges had budgets in the $150M range in the early 2010s [1] along with a separate $2B budget for CUNY's central organization (as mentioned above)
What seems to have happened is the above page treated overall TCV as part of a single budget, when in reality, each individual constituent org with it's own budget spent out of their own pocket for ERP, HRM, Headcounr (PS and internal), and Compute (on-prem)
This isn't egregious for the size of organizations that each of the constituent colleges and CCs in CUNY are.
The big question is, if you used the PSC's (imo flawed) methodology, what would it give for CSUs, UCs, and SUNY (all of whom have a similar structure and student size). I'd expect a similar amount.
[0] - https://www.cuny.edu/wp-content/uploads/sites/4/page-assets/...
[1] - https://www.ccny.cuny.edu/sites/default/files/finance/upload...
This should be the top reply. Thanks for digging into it.
On top of that, digging into their manifesto in the page above, I found this startling and horrifying stance the PSC has
> CUNYFirst will be part of the arsenal by which CUNY Central shoves Pathways down our throat
Basically, the CUNY PSC is opposed to a unified core curriculum, allowing students at all CUNY institutions to transfer between each other.
This is something that is diametrically opposed to student well-being.
Basically, the PSC at the time opposed NYC community college students from taking courses at CUNY's 4 year universities and vice versa, and was strongly opposed to interoperability and transfer of credits.
California's equivalent system (ASSIST) has had a massive benefit in allowing Californian community college students to transfer to CSUs, UCs, and private schools like USC and Stanford.
CUNY's faculty union (PSC) on the other hand is completely opposed to a similar system being adopted.
The whole point CUNY PSC was opposed to the CUNYFirst system was basically because the PSC was completely opposed to allows simplified inter-college transfer.
All things are possible. Here’s a scandal where the NYC government spent $700M to implement a time card system. CUNY has access to public authorities like the research foundation, which may fund stuff in addition to their appropriations.
https://en.m.wikipedia.org/wiki/CityTime_payroll_scandal
If you go to openbookny, a contract website run by the NYS Comptroller, CUNY has a $800M Oracle contract that has dispersed $630M.
This one for $800 thousand?
https://wwe2.osc.state.ny.us/transparency/contracts/contract...
Whoops. Mea culpa! That’s what I get for quick takes on a bus.
> Organizations typically add 3-5x additional to either "pad" their request (in the event it doesn't get 'fully funded') and/or this is an opportunity for the university to higher for a bunch of roles they wouldn't have been able to get funded in the first place - so lots of things get buried in these numbers.
A factor of 3-5x seems insanely overboard. Thats arguably fraudulent. Ditto for claiming the money is for one thing and then using it for something entirely different.
I get that it’s a communist era style scraping to get by / work around a broken budgetary system life in academia, but that’s misappropriation of tax and tuition dollars.
You ask for 5x the need, get 2x the original amount, and then when the project costs 1.9x as much as expected you're still slightly under budget!
Reminds me of an anecdote about a lead engineer for metro rail projects in India.
The lead engineer has over-designed the infrastructure by 3x over the standard. The junior engineer thought he was incompetent.
Turns out, he's paid the big bucks to add affordances as counterbalance for corners cut due to corruption in every step of the production line.
Yeah seems like 50-100 % administrative overhead for each employee, hard to believe that someone would accept that, you could probably pay each employee a personal valet for that.
50% overhead is pretty common for academic institutions. For example, indirect costs for an awarded grant are often 50%+.
And yes, it's ridiculous.
I've heard some of the national labs have 100-200% overhead for some grants
I'm reminded of the Oregon state health care exchange fiasco where Oracle took like $250M and never delivered a working product. Ended up in a lawsuit and a settlement.
It was also reported by much more reliable sources, and was a statewide project.
[deleted]
This is not a particularly useful contribution.
Source 1: Only mentions anonymous sources. And while I would put partial weight on that coming from an actual journalist, this is the same professor who wrote the original blogpost.
Source 2: Same publication as Source 1, though I don't see a byline. No named sources.
Source 3: No named sources. The "About Us" features Lorem Ipsum and four photos of the same woman.
Source 4: A reddit post, no sources, and the amount quoted is not $600 million, but $250 million.
Source 5: A petition. References an Oracle press release that didn't show up in your Google search (most likely because it doesn't exist, or it would be widely cited).
Having worked for NY gov agencies for 7+ years, it's not that hard to believe really. Vendors overcharge crazy amounts all the time and get away with it without the gov client batting an eye. Experienced it first hand where I was on several projects where I did, without exaggeration, 95% of the work and my employer at the time charged $500,000 for one project and for another $750,000 in a single fiscal year.
Sure, it wasn't $600 million (seems like that $600 million to Oracle had to of been spread out too), but the work was simply formatting their website content from an old template in their busted CMS (pretty sure it was some ancient HP CMS) in HTML and had to write some minor CSS and some minor JS. I ended up automating it locally and then the rest of the time was spent copying and pasting through this convoluted process due to having to connect to their outdated locked down remote desktop with a super slow connection. Additionally, I was only being paid under 6 figures at the time and it took much less time than they were quoted for, though my employer sat on their hands to make it seem like it took that much time.
But yeah what the agencies were charged was unjustified irl... but on paper it was, they didn't push back on it or anything, and the cost of completing the projects was wayyy the hell less than what my employer at the time quoted them for. My assumption is they dont push back due to budgets not rolling over and wanting to maintain/increase their dept's annual budget. So it's not surprising they'd throw away that amount over the course of some years.
When you see how badly most academics and academic administrators are at actually running things from a business standpoint it no surprise the academic sector is in such a mess at the moment. Sadly this whole mess is funded by debt handed out that’s not dischargeable in bankruptcy for degrees of highly questionable value. It’s really sad when you follow the money and think about who and how stuff like this is actually paid for.
The whole thing is kept alive by the student loan program. Modify that or take it away and academia in the US would implode.
When you see how badly most business administrators are at actually running things from a business standpoint it no surprise the business sector is in such a mess at the moment.
If you've ever worked in a large corporation or government, you know that this is the norm. It's amazing that anything, anywhere, ever gets done at all in organizations larger than a few hundred people.
We as humans are very bad at this.
Companies like Oracle, Deloitte, McKinsey, etc. are experts at extracting large sums of money from large dysfunctional organizations.
I've worked in academia. I've worked in small, medium and large private companies and fortune 50 companies, and I've worked for small and large govt.
I have yet to see anything bigger than a small company run efficiently, and that's by necessity more than anything. The largest wastes of money I've seen were fortune 50/large enterprise companies yet people constantly point to gov and academia as the most wasteful.
My current job is a cycle of hype waves the executives and ceo buy into and oversell to customers in impossible timelines, the inevitable smack of reality when neither CEO nor customers are happy with the over-promised and under-delivered results, and the move to the next wave to capture more customers to replace the ones attriting because the result doesn't match the requirements despite waht they were told.
The actual users of our product seem to split fairly evenly on love or hate our products with good reasons while their executives and spending managers are almost comically willing to fork over more money over for little to no real benefits. There is very much "No one was fired for buying IBM" mentality in the corporate world, this seems like another example.
> Companies like Oracle, Deloitte, McKinsey, etc. are experts at extracting large sums of money from large dysfunctional organizations.
And IBM and Accenture, and etc etc
Feels like this entire sector *should be* ripe for disruption by a more skilled and far leaner organization for far cheaper, but my impression is that the majority of these contracts are some "business/partner" relationship things, and those that opt for these contracts are performing CYA by opting for these companies.
I've worked in a large government organization (the IRS) and at a large tech company.
I think it is fun to pretend like the issues in private industry vs public sector are similar in magnitude, but in reality it's not even close. I have never seen even remotely the level of dysfunction I saw in the public sphere in any private company.
Based on your comment, you've worked in 1 public and 1 private organization. Not sure this is the basis for coming to such sweeping and drastic conclusions.
We're all working from limited examples - this entire article is about only one. I know many other government workers in different branches who felt similarly.
and yet you discount even the commenters in this subthread who report on the horrors within the private sector ....
> Companies like Oracle, Deloitte, McKinsey, etc. are experts at extracting large sums of money from large dysfunctional organizations.
I wonder, how do they do it? How do they sell sub-par products/services that a company arguably doesn't need at a premium price?
Lot of people don't care about spending someone else's money efficiently. And in some cases spending more will make them look better and lead to better opportunities. And this happens on every level, lot of employees just don't care. And those that might probably focus on wrong things.
And no software people are no better, going for expensive tools, expensive cloud spending or just next shiny thing is often argued with some time to market or future scaling excuse. Or just wanting to do something different.
Are they themselves large, dysfunctional organizations?
Why does no one on the inside even seem to acknoledge is publicly (I've seen enough internal discussions in the private sector to know they do complain on the inside)? Let alone try to fix the dysfunction? Does someone high up have perverse incentive or are simply too conservative to risk chipping away at the problem?
I'm sorry but having worked in private and public sector, there just really isn't any competition in terms of incompetence.
If you're not exposed to market forces....
And yet academia manages to be worse.
Citation?
Ineffective corporation will eventually be driven under the ground by various forces, protected businesses have 0 pressure for quality and effectiveness, so unsurprisingly there is little to none.
Anybody who dealt with good amounts of state bureaucrats can see this rule in plain sight. Bonus points if you actually know some privately. Then they have the balls to want same lifestyle as somebody working hard on themselves and their careers, doing sacrifices they wouldn't even dream of accepting. Of course success never materializes, that evil capitalist world is against them.
I recently left a software dev job in academia, and the amount of inefficiency in the organization is insane.
As part of my off-boarding, the lead engineer mentioned she would ideally like to hire 5 more developers to the team. That would bring the team size up to 15 (8 devs, 2 devops, 2 Ux, 1 graphic designer, 1 pm, 1 eng manager). The team maintains two things:
- The static website for the library
- A pretty basic image server and viewer for the library and museum collections
Sure, the library needs a website, but that shouldn't require more than a person or two to maintain. The image viewer was only used by a handful of people.
But it doesn't matter. The team gets funded because the students will keep paying their tuition. The engineers can keep sitting around watching youtube all day and the world keeps turning.
Perhaps most egregious example was in my first 1:1 -- my manager said, "Don't expect much output from [SENIOR ENGINEER X]. He isn't a good engineer." The organization isn't willing to fire anyone. As a result, the people in charge are the ones who have been around the longest.
That said, it's risky to fire anybody because hiring is so difficult. The salary bands are set at the university level for all employees, which means the maximum a software engineer can make is much lower than market rate. To make matters worse, working "in person" is mandated by the library dean, and the university is in a college town in the middle of nowhere. The interview process also includes NO coding sessions although I'm not sure whether that's due to some corporate process or just incompetence.
To be fair, these sorts of issues aren't exclusive to academia, and I've seen similar issues in large organizations. Somewhat paradoxically, the more bullet-proof the business-model is, the more potential there is for rot to grow in a company.
> Perhaps most egregious example was in my first 1:1 -- my manager said, "Don't expect much output from [SENIOR ENGINEER X]. He isn't a good engineer." The organization isn't willing to fire anyone. As a result, the people in charge are the ones who have been around the longest.
Kidding aside, where can I find one of these jobs? I'm burnt the hell out and would love to lick my proverbial wounds for a year or so doing this.
I'd love to get a job like this seems amazing. Imagine all the gold plating you'd get to do. All the experiments and optimizations. You'd have the best library website ever.
Imagine the job security.
When you're on a team with 15 people, that means a lot of process to push through to get anything done. One of the other engineers was the owner for analytics, and as a result we had basically no analytics. Furthermore, the team lead would often say things like "developers aren't designers and shouldn't be thinking about the product". I wanted to implement a basic semantic search to help people find the resources they were looking for (#1 pain point from talking to people), but that was shut down at every turn.
There actually was talk of firing someone once, but the reason was he didn't always arrive at the office at 9am sharp, and wasn't in any way related to his output.
They hiring?
This doesn't sound like the academics', or even administrators' fault- this is being imposed by the state university system. Reading between the lines the change is a response to political pressure to reduce costs and impose restrictions on curricula.
>> Reading between the lines the change is a response to political pressure to reduce costs and impose restrictions on curricula.
Reduce costs by spending huge amounts of money to lose capability? This smells like someone got an "incentive" to spend public (govt) money on some corporate project. Not sure why anyone wants to impose restrictions on curricula, but that'd be a kind of separate thing.
>Not sure why anyone wants to impose restrictions on curricula
I won't get too political, but that's the one part of this story that only got worse and more blatant over the decade since this posting.
I'm an academic at a large state university (not SUNY). Faculty, staff, and students generally have very little direct say in IT-related matters. These decisions often come down from central admin, through a process that is just as mysterious (and sometimes infuriating) to us.
I will also chime in. Having contracted with multiple large, state universities, this is the norm. Staff and faculty have little, if any, input into the systems the university uses, and are often just as confused as the rest of us.
Every institution I've worked for had a check-off for IT and central admin if software purchases were requested. These are well-known to be poison to most initiatives without a Dean level or above pushing for it.
I don't think people that criticize non-admin university employees have any idea how these things work. Not only do they not talk to the people that do the work to find out what they need, they're not open to feedback on the garbage they've cobbled together after they put it in production, and every decision is made assuming faculty, staff, and students are always wrong and they're always right. I could write a book about the things I've seen.
Nit: SUNY and CUNY are surprisingly unrelated.
I had to look it up, and I live upstate.
The university I was at had this amazing process for internal billing. Everything cost heaps and required entry in an arcane system. If you could do something inhouse, you were supposed to, either though the cost was astronomical. A $100k bit of software from an extra vendor was definitely $200k+ once the various IT departments (there were at least 4) had done their bit. There were managers all the way down and all were very important.
I have found that most academics have zero interest in how the sausage is made. In fact, I would say there is a willful ignorance to understanding the intricacies and complexities of what it takes to run a University system and the precarious financial realities of most colleges and universities.
However, every once in a while an academic decides to take on administrative responsibilities to fix all of the things they perceive as broken or to show everyone how smart / right they are. Usually the first year for them is incredibly difficult for them and everyone around them and they make a true and terrible mess of it. At this point, after a full year, they usually know just enough to realize how little they know about running a college, managing people, or generally being a leader. They then react in one of three ways:
1. Resign from their admin job and go back to just teaching as if nothing happened
2. Become humble and begin actually collaborating with people and not blaming everyone for every thing that isn’t as they see it.
3. Double down and truly blow everything up until they are either fired or whatever it is they are running collapses in on itself.
This isn’t everyone. The faculty who transition best to leadership tend to be pretty humble to begin with.
A bunch of years ago, at college, I built a class management platform for my university. It won me $1k (which was an insane amount of money to college me), and I met with president of the univeristy to pitch him on using it. They were potentially buying Oracle software, and I found myself up against them. At the time, I felt like this professor.
They obviously went with Oracle. I'm sure they spent a ton of money, and I'm sure it was the right choice. I would have gotten bored of it pretty quick. You don't pay Oracle because it's a good deal or a good product... you pay Oracle so you never have to think about it ever again.
I don't really have a point here. I wish there were better options in the market. But I certainly don't want to build them, since it's a boring problem with bad customers (edtech is a horrible sector to sell to). Oracle has a price point that makes it worth it to them, and they have customers willing to pay it.
Hopefully someone sees this blog, and rather than be annoyed at academic/government waste, sees a big market they can dominate with a better product. But given how it was written in 2013, I'm not so sure.
They're also paying $5M per year to maintain the servers and software. In theory they don't have to think about it but that just highlights the waste even more. I wish I could spend $5M without thinking.
CUNY's yearly budget is $5.7 billion, and they serve 243,000 students. It makes up 0.088% of their budget.
That $5M/year works out to $20/student/year, so in perspective it's a very small amount of money.
(Source: https://www.budget.ny.gov/pubs/archive/fy24/ex/agencies/appr...)
> so in perspective it's a very small amount of money
$5M/year could pay for at least another 10-20 professors which is like an entirely new department. Let's also not forget the initial $600M spent which (assuming they'll use this software for the next 30 years) will be $20M/year bringing the total to $100/student/year which isn't negligible.
More broadly speaking, I have a gripe with people bringing up percentages of budgets when discussing how much value something brings. My college had a startup fair where students pitched some super innovative ideas. The grand prize was $5000. If we had $5M, we could give that grand prize to 1000 teams and imagine how much more valuable that would be for the school compared to another feature bloated, overpriced piece of software.
Okay, so if you hire 10-20 more professors... who's going to build and maintain the system so students can sign up for those classes?
Programmers. You code the system once, it's not that complex, spend $10M max.
Then you maintain it with a couple of senior devs at $1M/year max.
The one-letter omission of "k" in "$20k/student/year" really makes a difference here.
No, $5M / 250,000 is $20. (I updated my comment to add a clause about $5M, to reduce confusion.)
> edtech is a horrible sector to sell to). Oracle has a price point that makes it worth it to them, and they have customers willing to pay it.
The way businesses deal with this is by selling more expensive software! That's why Oracle is getting rich by selling crappy software to difficult sectors that nobody wants to touch.
Well sure if you can manifest money like a daydream, then you won't have to think about it anymore. I don't know though, maybe there are some folks out there who do in fact spend time thinking about unnecessary costs?
Yes, as an individual you do. But you're not a network of universities with 250,000 students a year that NEEDS the system to work.
(Also, most people don't agree they're paying Oracle $600M for this... there's no source, and it's likely significantly less.)
My understanding is that this network of universities is in fact a bunch of individuals who sometimes care about costs they are generating for their customers.
@dang — I found a better link which appears to be a revised version of the current one: https://psc-cuny.org/clarion/2013/may/cunyfirst-users-last/
---
This is a post by Prof. David Arnow on the blog of the Brooklyn College’s professors’ union about CUNYfirst, a PeopleSoft-based course registration and HR system sold by Oracle. Posting because the system recently got some attention on Twitter: https://x.com/ChocolateyCrepe/status/1836171439965446441
It seems like there should be 5-6 vendors of 'University HR Software'
You want 1,000 licenses for it? That will be $5,000 a year, for a total of $5 million
It's going to us a year to implement it, we're going to send out 25 people to get it up and running, train your users, etc. That's another $25 million.
We'll spend the next year building integrations to all of your other software and systems, that's going to be another $25 million.
These quotes will all vary +/- 25% depending on the vendor. Schedule a 200 hours of meetings, trainings, etc for 500 people involved with the new software. That's another $5 million.
Where's the other $540 million coming from?
>> It seems like there should be 5-6 vendors of 'University HR Software'
Why can't the CS department join up with other universities and have a bunch of students build it? Open source it.
Well CS departments are rarely staffed by people who are interested in building actual software (if they were, they could just work in industry making significantly more money), and HR software is probably the worst vertical for a bunch of kids with no experience working in.
My school tried this in the early 90s: they held a contest to make class scheduling software. I don't think it worked out- I certainly did not enter it.
This was to replace the previous system: professor John van Alstyne just did it over a weekend.
I think that's a fine idea for a trade school
I worked in IT at a CUNY school at the time. Just hilariously poor and unintuitive. Every student was given an Employee ID number. Course registration was basically handled by an e-commerce addon.
Sure beats what SUNY was doing a decade earlier. Your student ID number was your SSI and it was printed on your ID along with your name and picture.
IDs were often lost and stolen.
Sounds better than a $600M solution to me.
This was the $600M solution..
Ouch, I thought OP must’ve been talking about before the Oracle system.
That was the $600M solution...
If you think about, $600M is enough money that someone somewhere could create an entire new company and staff it with some of the best developers just to get this contract...
Or CUNY could hire developers at silicon valley salary and build this in-house.
As other posts have pointed out, it is unclear the 600M figure is accurate.
And if this was the up-front cost, I'm sure there would be more DD about alternatives.
But I'm sure the lock-in occurred long before the bill hit 9 figures.
Regarding the other bidders dropping out, what exactly was the primary expense that gave it an estimated $1B total cost? $600M can build out a greenfield software platform, so there must be more to it.
Garbage overpriced systems that everyone hates, and which substantially hurt the organization... How do you land those sales? (Joke: "Asking for a friend.")
Some bad ways that big-ticket purchasing decisions are made:
* Committee of people who don't know what they're doing, and/or who can't coordinate to arrive at a good holistic decision/solution.
* Person who wants this done for good reasons, but doesn't know what they're doing.
* Person who wants this as an accomplishment credited to them, but doesn't know what they're doing.
* Person who is mainly thinking "nobody ever got fired for buying [old big-name vendor]", and everything else is secondary.
* Person who is bribed by vendor (e.g., immediate cash, quasi dates with attractive salesperson, career rotating door with vendor).
Other ways?
(I haven't directly seen the bribery way, though heard of it in news stories. I've definitely seen all the other bad ways happen.)
1. Be willing to spend 5 and 6 figures (unreimbursed) responding to lengthy RFPs. 2. Work with an established Systems Integrator (who will take a large percentage.)
A $600M Anti-Chesterton Fence that nobody knows why it’s there and everyone’s scared to touch.
In government procurement positions, I would be in favor of giving a small %-ge (to the tune of a few hundred thousands of dollars) of money saved from budget to the people in charge of procurement.
The California State University system contracted with PeopleSoft to consolidate their various non integrated systems back in 2003. Initial costs were supposed to be 350 million, but instead ballooned to over 700 million. We still don't use the full functionality of the suite, even though it's all been payed for.
Full report here: https://information.auditor.ca.gov/pdfs/sr2004/2002-110.pdf
And it's the absolutely worst system ever, a complete pain in the ass in every way. If you've never touched PeopleSoft, they not only spent hundreds of millions on a system, but on a system that sucks away years from your life-force.
So I have an idea...
Why not let the computer science students take a crack at building some of those systems? Maybe not the an HR system, but why not a course registration system? Maybe not for the whole university, but maybe for just the computer science department?
The university would get work for free and the students would get real-world practice with building production code.
I get that there would be risk, but if it was under the supervision of professors (who hopefully are good at building, not just lecturing theory), I think there is an opportunity there.
> I get that there would be risk, but if it was under the supervision of professors (who hopefully are good at building, not just lecturing theory) […]
Good joke here! If you are actually serious, please tell us which university you encountered where the majority of professors actually did something productive in computer science
Personally, I would not paint all professors with the same shit-stained brush.
Sorry if that came around as rude. From personal experience I have not met any professor in my life who could lead such a project, let alone with students that don’t have any work experience… And you imply there should be multiple of them for the project to succeed
Someone has to maintain the code. Can you imagine making students doing that kind of work? They'd rightly asked to be paid for it. And getting faculty to manage the students doing that kind of work, they'd definitely want extra pay.
>Why not let the computer science students take a crack at building some of those systems?
they need someone to blame if something gone wrong.
This mentality needs to die.
I'm generally a Hanlon’s razor advocate. Never attribute to malice what can be sufficiently explained by incompetence.
With that said this example strains even my ability to justify the incompetence angle.
Hanlon's razor is significantly dulled by the presence of Larry Ellison.
Perhaps I can help... when Hanlon's Razor doesn't explain the malice/stupidity in play, you can always try the extending it with the following:
Hubbard's corollary to Hanlon's Razor: "Never attribute to malice or stupidity that which can be explained by moderately rational individuals following incentives in a complex system". (https://en.m.wikipedia.org/wiki/Hanlon's_razor#Other_variati...
Or HN Nerdponx's simplification: "When money is at stake, never attribute to incompetence what could be attributed to greed." (https://news.ycombinator.com/item?id=41066724)
Probably a kernel of malice (corruption, kickbacks) surrounded by a warm flesh of incompetence.
Grey's Law: "Any sufficiently advanced incompetence is indistinguishable from malice."
>Oracle-PeopleSoft did not drop out. However they warned CUNY that for that level of funding, they could not, would not CUSTOMIZE: they would only CONFIGURE
In this case, I don't think Oracle is to blame.
They explicitly warned about the 'limitation,' yet the project proceeded nonetheless.
In my opinion, for this project to succeed, it should have been built from scratch. With the $800 million invested, they could have assembled a team of seasoned and junior developers to get it done. Just my two cents.
I find it really hard to justify spending $600M on a system like that? Look, if you had 500 skilled FTEs working on the project for 2 years, at $250K per FTE, that would be a total cost of $250M. Say 50 FTEs for ongoing support at $12.5M/year or $125M for 10 years. So $375M.
But the above numbers are hugely generous. This is not building an ERP from scratch. Do you really think it would take 500 people (say 400 engineers and 100 non-engineers), to build and deploy such a system? I would imagine you could get it done (and done right) with half that many, or less.
Anyway, just mind-blowing.
IF you go to a body shop to get a dent pulled, you will receive one quote. If you tell them insurance is paying you will receive a second, much higher, quote.
The dealer will say it's because the insurance company is such a hassle to deal with that they have to dedicate employee time to jumping through hoops to get paid. The insurance company will say the quote is higher because the dealer sees the insurance company as a big bag of money and if the insurance company doesn't ride them then prices of insurance will go up.
Dealing with government is this times a thousand.
Your analogy is somewhat backwards. Insurance companies squeeze body shops pretty well. If you go in there and ask for the same work you will pay more.
The real difference is that often times a person will just say make it look okay and I'll be fine. Whereas insurance work is basically making a car good as new because that is legally required. If you try to get the same level of service at retail you will pay more.
Interesting. I've been lucky in that I haven't had non-cosmetic damage repaired, but it always seemed like there were several iterations when insurance was involved.
There are tons of body shops that won't even take insurance work because of the low pay and somewhat low standards of work as well.
With Oracle? 400 Engineers is likely way to high, it's more like 100 Engineers and 900 non-engineers. Oracle likes to throw in a ton of people into contracts with lie that they will provide "white glove" teaching and various other things.
I would say you probably could get it done with 5 people (1% not 50%)
Maybe not 5, but fewer than 50, yes, even with QA/testing/deployment. So long as they're skilled.
5 is a stretch for sure, but Retool, for example, grew to a 9 figure valuation with 2 engineers, and AI is now a big boost to developer velocity.
It might be reasonable, if it was quality software. But it is crap.
Governments should unite to create basic open source software and then individual organizations can tailor it to their needs.
The trick is the find some way that a vendor can monetize it. Sometimes you have management consulting groups building crappy unmaintained (except by them, if you extend the contract) OSS.
One thing that seems to work, sometimes, is having cloud vendor support. Could you support maintenance of an OSS platform that AWS (and/or their competitors) operates - that way AWS could "win" the contract, but the OSS system gets funded with a core team and various groups still contribute to make it better
I agree. Any tax payer money used for creating software should be required to be open source. Ideally, governments and similar organizations would utilize open source software, then use contractors to modify, support, and maintain it (again, releasing any changes as open source).
Contractors would then be responsible for providing excellent support, not some huge bloated product.
It is a partisan fight for the government to build and offer tax software. Huge uphill battle to build out infrastructure code projects.
Governments are controlled by rich people's interest. Of course they are lobbied to death to avoid any such kind of solution.
Bribery isn't control. It's just bribery.
Direct or indirect, it doesn't matter. It is a strong form of control.
Pretending it is impossible to get a semi-democratic government to work for the people instead of just very difficult is defeatism. In fact, getting people to believe "It's hopeless" is a big part of the way bad governments maintain control of people, like in Russia or in the USA
I'm not saying it is impossible, I'm only talking about the situation in the US. Currently it is dominated by the billionaire tech and financial oligarchs and we need to do something to change it. It doesn't make any good to sugarcoat this reality.
Governments aren't able to manage their way out of a wet paper bag unless it threatens their existence.
And yet 90% of all startups fail, which could also be seen as a indicating a certain level of incompetence in the private sector.
Consider perhaps that incompetence is distributed across the economy, and government and private industry share in that, with successful and unsuccessful projects as a consequence.
Governments can print themselves a bigger budget. Startups can't.
Pro Tip: If they're willing to sell it, you don't want it.
Drexel University nearly bankrupted itself in the 80s, among other things, by hiring Lockheed (then Martin Marietta) for a similar boondoggle project. The project was an unmitigated disaster that never delivered anything, and the university had to pay to exit the contract.
Wow, if I’d known about that contract I’d have bid $500 million. I could hire some folks and get that done.
How else do you expect Larry to afford control over surveillance state?
UOH!
crying emoji
My goodness, the world never changes. Saw the same "new enterprise software much worse than the incumbent, terrible UI/UX, everyone hates it" dynamic play out *30 years ago* with SAP R/3. When I check back in 2054, expect it will not be any different. Technology changes, but people and organizational dynamics largely do not.
As a somewhat related PeopleSoft story, the government of Canada paid IBM and a variety of contractors $3.5B over several years for a PeopleSoft-based payroll system called Phoenix.
The system was a disaster. It never worked properly. So the government is throwing hundreds of millions towards its replacement.
to manage means to extract-value-from and $600M for software is a lot of value extracted.
I'd bet huge that there is a layer of managers who don't see themselves as being accountable for domain competence in anything they manage as that's what the consultants are for. Consultants mean headcount and budget to manage- which is the definition of success in an institution. they run an organiation, the mission is little people IC problems.
It's not broken, it just works for people you can't see.
>> I'd bet huge that there is a layer of managers who don't see themselves as being accountable for domain competence in anything they manage as that's what the consultants are for.
Ah the old "management is a generic function" argument. A CEO from Coke is qualified to run GM or Intel. Just have the executive team give me all the information and spell out what results in the biggest $$$ and I'll "make the decision". I thought that silly notion was sunset some time ago...
it's how institutions work and it's very alive anywhere with inclusion programs of any kind. to hire managers based to anything other than strict hierarchical domain competence depends on the premise that managers are generic low agency minders of things that are already creating value themselves, and a transferrable/interchangeable function. nothing outdated about that at all.
Does anyone know what "pathways" is and why they wanted so much central control to set that up? It seems that there was some sort of power angle to all this.
Bryan Cantrill's rant about Oracle for those who haven't seen it:
https://www.youtube.com/watch?v=-zRN7XLCRhc&t=2047s
> There has been no entity in human history with less complexity or nuance to it than Oracle. [...] This company is about one man, his ego, and what he wants to inflict on humanity. That's it.
And about society's willingness to tolerate such a man.
With that money, Larry begged Jensen to take it and deliver him the GPUs. Onto the new grift. In yesteryears it was database and now it is AI. Benedict Evans always compared AI to databases. Oracle of AI is THE Oracle of Databases.
It's worth remembering that Oracle bought Cerner, a major EMR vendor, and that they plan to rewrite Cerner Millennium (the EMR) using AI.
Enjoy your hospital stay.
I have been negative on Oracle since the ‘90s, when I had to work with an installer CD for Oracle software —- an official Oracle CD —- that:
1. Had multiple installer applications on it, with no indication which was “the” installer application. 2. On opening the installer, asked me to select the install file to act on, again with no clear direction. 3. Had “help” files on the disc, in HTML format, which contained broken links to other files on the disc.
At the same time my coworkers, experienced Oracle DBAs and developers, with full paid-for support from Oracle, spent an entire summer trying to install an Oracle development environment, and failed.
All of which to say, yeah — $600 million sounds about right, as long as it turns out the software was never successfully implemented.
Now that's a... difficult company name.
Certainly one of the names of all times.
Was it written in PHP?
Every time something comes up, this is how I explain it to people:
"Why Enterprise Software Sucks" https://x.com/random_walker/status/1182635589604171776?lang=...
Here is the unrolled tweet for those without an active account:
https://unrollnow.com/status/1182635589604171776
You the real MVP. I wish Twitter users would understand how awful the platform is for someone without an account.
I have an account and I think Twitter is extremely awful.
I never post anything, I just have the account to open links people sent me.
This is why I never, ever, buy or recommend software without actually using it along with the people who’ll be dealing with it everyday. “Oh, devs will have to install it on their laptops? Ok! I’m a dev. Send me a test license and I’ll see what it’s like.”
I’ve made mistakes, sure, but I’m proud to say I’ve never signed off on anything I wouldn’t be personally willing to endure.
The baby clothes analogy is perfect!
And it works just the same for Workday and ServiceNow applications. And TV remote controls...
Don't forget Salesforce!
here's the prior HN post about that tweet: https://news.ycombinator.com/item?id=21224209
In brief, administrators are tunnel-visioned on checking boxes without being mindful of the fact that if you only need 1% of the features, the other 99% represents unnecessary complexity that will directly impact the usability and maintainability of the "solution".