Thoughts about the role of Chief Architect

Marc Adler, CTO as a Service

The following observations about the role of Chief Architect come from my four previous positions as Chief Architect. I have been Chief Architect of the Equities Division of Citigroup (350,000 in the company, 30,000 people in Equities), of MetLife (65,000 employees), ADP (65,000 employees), and of a small software vendor named Quantifi (roughly 50 employees). I have a mixture of large company and small company experience that has shaped my perceptions. In addition, I owned my own small software business for roughly ten years, and that experience has also shaped my thinking.

I may be slightly biased in my thinking, but in my opinion, the role of Chief Architect is one of the hardest roles out there, both in terms of defining the role and in terms of what is expected of you. Each company has their own idea of what a Chief Architect’s role is, and some companies don’t even know why they need a Chief Architect.

The role of Chief Architect is an extremely “vertical” one. I always like to say that a Chief Architect has to be able to  interact at the CxO level, and be able to talk to the lowest-level coder. The Chief Architect has to be comfortable in executive-level meetings, and must be equally comfortable sitting next to a developer and doing pair-programming or code reviews. Often times, the Chief Architect is called on to explain technology to the CxO-level people. The Chief Architect is often called on to face off with the senior-level technical people at vendors and partners. I have been directly opposite CIOs and CTOs of vendors on many occasions. The Chief Architect should know the business and well as the technology.

In my opinion, there should really be only a single Chief Architect in a division or an enterprise. There should be a single point from where all architectural decisions are made. This jives with the list of responsibilities that a Chief Architect should have (see the list below). If there are multiple Chief Architects, then there is more room for disagreements and to get into “analysis paralysis” mode.

Management

Many times, the Chief Architect has to manage a team of architects. At Citigroup and MetLife, I managed multiple groups which fell under the Architecture organization, and these groups included performance optimization engineers, project managers, and business analysts.

Since I have typically managed teams of architects, I have had to slice my architects into different domains. For example, at Citigroup, I had an architect responsible for Cash Trading Systems, an architect responsible for Derivatives Trading, an architect responsible for Risk, etc. I also had architects devoted to SDLC, architects devotes to performance optimization, etc. Each architect was responsible for a vertical slice of the entire organization, but the architects had to work horizontally as well.

Since the Chief Architect was regarded as a senior or area manager, the Chief Architect often reported to the CIO or CTO of a division. The Chief Architect would often be the CxO’s “right-hand man” when it came to technology. It was unimaginable that a Chief Architect would report to a development manager, since part of the architect’s responsibility was guiding development practices and doing architecture and code reviews (something that would sometimes put the development manager and architect at odds).

Hands-On Participation

A constant question revolves around how hands-on the Chief Architect should be. I advocate for a hands-on architect for various reasons.

The Chief Architect should not be known as a “yellow pad architect” if the CA is to get the respect of the development organization. The developers want to know that the Chief Architect has walked in their shoes before they accept his advice. A Chief Architect should be able to speak to any CxO or sit down and pair-program with the lowest-level developer.

Since the Chief Architect is also responsible for exploring new technologies and delivering proof-of-concepts, the Chief Architect should be able to code these POCs personally.

Therefore, I strongly advocate that the role of Chief Architect not prohibit the CA from diving into a coding role on occasions.

Roles and Responsibilities

The list below is a union of all of the responsibilities that I have had as Chief Architect.

  • Manage the Architecture Organization
  • Attend meetings with current and potential vendors
    • Be able to face off with the CTO or Chief Architect of a vendor and do technical vetting
  • Evaluate vendors and technology using an Architecture-team-developed scorecard
  • Keep abreast of new technologies that might be beneficial to the organization
    • Run the Innovation Lab
    • Meet with vendors
    • Do proof-of-concepts, often involving coding from me or a member of my architecture team
    • Attend industry conferences in order to be able to see what new technologies are hot, and to even see what our competitors are doing.
    • Occasionally give talks or sit on panels at conferences
  • Software development
    • Sometimes, the architecture organization will develop a POC or MVP in order to prove out ideas or to relieve pressure on the normal development organization
  • Give architectural approval for “big ticket” projects
  • Attend executive meetings
    • Often called on to decipher technology or render opinions on technology
  • Liaise directly with senior business stakeholders
    • Find out what kinds of business problems the stakeholders wanted to solve and propose solutions based on their needs
  • Guidance
    • Come up with reference architectures
    • Serve as a general technical resource for the development organization
  • Roadmaps
    • Provide roadmaps that show how the organization will move towards a certain future state
  • SDLC process
    • Do Architecture and Code Reviews
    • Attend Sprint kickoff and retrospectives
    • Provide approvals for code to be deployed into production
    • Advice on coding practices
  • Performance Optimization
    • Help the engineering team with the optimization of certain systems
  • Governance
    • Ensure that the organization is using the appropriate technology and software
    • Ensure that the organization is not using End-of-Life software
  • Architecture Review Boards
    • Run or participate in architecture reviews and approvals
  • Socialization of architecture across the enterprise
    • In the instances where I have been CA of a specific division, I would make the effort to find out what other CAs in other divisions were up to

AWS CodeBuild and Access to RDS

One of my clients whose application runs on AWS had no Continuous Integration (CI). The code is stored on Github, and I had just gotten the developers to write Unit Tests and Integration Tests. There are different tools that you can use for CI, including Jenkins, Travis CI, Circle CI, and more. But, since my client is a heavy user of AWS, I wanted to try AWS CodeBuild, as it seems to be tightly integrated with a lot of other AWS PaaS products.

I set up CodeBuild to pull the code from Github, and to run the Integration Tests using knex mocks. Everything worked smoothly.

The next step was to set up a Postgres/RDS database that was devoted to CI testing, and to switch from using mocks to using a real database.

The problem was that the application code could not access RDS. All access from CodeBuild to RDS was blocked.

The solution that I came up with was as follows:

Note which AWS region your CodeBuild instance is running in. For example, mine is in us-east-1 (select the build and look at the details to find the region).

• In your browser, go to https://ip-ranges.amazonaws.com/ip-ranges.json, and look for the entry for CODEBUILD in the region mentioned above.

• Note the IP address associated with your region of CODEBUILD. For example, the IP address for my instance is 12.345.6.789/28. (Of course, this is a fictitious address)

• Now go to the RDS AWS Dashboard and find the instance of RDS that you want to access through CodeBuild.

• Find the Security Group that the instance of RDS is using

• Navigate to that Security Group

• Go to the Inbound Rules, and add a new rule for CodeBuild. I added a new TCP rule for 12.345.6.789/28, using port range 0-65535.

• Go back to CodeBuild and run the build. CodeBuild should be able to access Github (like before) and now it can access your private RDS instance.


The number of questions found on Google around CodeBuild users accessing RDS are such that I would think that the AWS team would make this into some kind of point-and-click visual interface.

Distributed Systems Meetup in NYC

This a part of the continuing education process that a good CTO/Chief Architect has to maintain. This is a photo from a regular Meetup that I attend, where we discuss Distributed Systems. In this particular event, we were discussing Fault Tolerance in Distributed Systems with our host, Andrew from Venmo.

For people who want to learn about the theory behind distributed systems:

I feel that you can get your Master’s Degree in CompSci solely by going to meetups every week. So much great information is exchanged, and you meet others who are using interesting technology in their day jobs.

Does a Large Software Project Need a Software Architect?

I just answer this question on Quora. I am happy to hear any comments that you might have about my answer.

My background is that I was the Chief Architect of 4 different companies (as now a CTO for Rent), so my answer is colored by my experiences and my biases.

There are all sorts of “architects” out there. Enterprise architects, solutions architects, business architects, software architects, infrastructure architects, etc. Everyone fancies themselves as an architect.

So, what distinguishes an Architect from an ordinary Developer, and why would you need one.

I am going to make an assumption that you are in a large company since you mentioned a “large software project”. I am going to assume that there are teams of developers, project managers, business analysts, etc, and that the project is budgeted at several million dollars.

In large companies, one of the jobs of the Architect is to make sure that what your team is developing is following certain standards. You do not want to be using end-of-life software, you may not want to be using some open source software that is only being developed and maintained by a single person, you want to use software that is a “buy” on the company’s buy/sell/hold list, you want to be using software that passes certain performance criteria, etc.

Another job of the Architect is to make sure that your project is well thought out. Are you thinking about scalability? About different kinds of failover? About certain edge cases? Is your team following good coding practices? Is the software extensible, so that new features can be added easily? Is it easy for other systems in the company to integrate with your platform? Does your application follow a future vision that the business has?

The Architect should be familiar with frameworks and patterns and lessons-learned that other areas of the company have developed, and should be able to recommend their use to you.

Often times, the Architect is closely allied with (or in charge of) the Common Services Team of your organization, so the Architect should be able to recommend frameworks that this central team has developed for the organization.

If your project did not have a good Architect, who would do these jobs for you? The Project Manager? The Product Manager? Both no. The Dev Lead? Maybe they would do some of these tasks, but the Dev Lead will probably live inside the silo of your team and not reach across the whole organization.

I hope this answers your question. Good luck with your project.

A Typical Meeting with a Potential Client

Some of you may be wondering what an initial meeting with CTO-as-a-Service looks like. Here is a typical the first meeting with a potential client, which usually occurs over a cup of coffee.

The potential client was recommended to me by someone in our mutual network. This potential client turned out to be the Poster Child for a CTO-as-a-Service engagement. She is a recent business school graduate that had a great idea (which happens to be aligned with a social cause that I believe in) and had some friends-and-family funding. She is not technical at all. A recommendation was made that she use a low-cost offshore development shop, and unfortunately, this development shop ended up not delivering what they promised and what she wanted.

She has some interest from potential investors, but she needs help in setting things right, and she needs help in setting the technical direction of the product so that it will be flexible and scalable as her vision expands.

As we chatted, it became very apparent that she needs someone like me, but her budget does not allow hiring an experienced CTO full-time.

I told her that is all good. CTO-as-a-Service can work with a client a few hours a week or a few hours a month. You could think of CaaS as you would think of your lawyer or your accountant. I can have weekly check-in calls with an outsourced development team in order to make sure that they are on target. I can draw up the initial technical architecture. You can drag me along to meetings with investors and tech incubators to show them that you have experienced leadership sitting behind your startup.

This potential client is going for more funding in January, and I certainly hope that she chooses me to help her realize her vision.

Free Syncfusion UI Toolkit for Startups

This news is of special interest to startups that are looking to build the first version of their product.

I just found out that Syncfusion, a very well-known manufacturer of UI Toolkits, has just released a free Community Version of all of their tools. The Community version is free to all companies and individuals with less than $1 million USD in annual gross revenue and 5 or fewer developers.

I have used Syncfusion tools for years, starting in the mid-2000’s when I was consulting for Wachovia and building a CRM system for their investment bank. Syncfusion provided a lot of UI components that we would have had to spend a lot of time writing ourselves.

The Community Version has UI components for ASP.NET (MVC and Core), plain old Javascript, WPF, WinForms, UWP, and Xamarin. I am especially looking forward to using the Xamarin components for an app that I am building that is targetting iOS and Android.

Case Study – Helping a Legal Software Company

TL;DR

  • Two week engagment (I can do one day to several month gigs)
  • Brought in to evaluate the current state and recommend a future state
  • The output was a 15-page detailed report, complete with a list of refactorings for the code and architecture, and a product roadmap
  • Client is now seeking approval to rewrite the platform

At CTO As A Service (CaaS), I will do on-site consulting, remote consulting, or I will even travel to your site. For this particular client, I was fortunate to have someone who was located in my home territory of New York City. All I needed to do was to take the M20 bus uptown.

People often wonder where you get your first clients from. In this case, it was from a great recruiter whom I have worked with over the past few years. One of her clients was a Private Equity Firm who took a stake in a company that provides Litigation Financing. An interesting aspect about being a “CTO-for-Rent” and parachuting into different clients is that you end up in domains that you didn’t even know existed.

Litigation Financing is an interesting field, one that is growing very fast. The gist of Litigation Financing is that a company will provide financing to people who are injured in accidents while their lawsuits are active. Often times, these people do not have disability insurance, and because of these accidents, cannot work for a period of time. They need money to live on. A Litigation Financing company will give (not lend) money to these injured people, and when the case is won or settled out-of-court, the money is repaid with a certain added percentage on top. If the lawsuit is not won, then the Litigation Financing company loses money. This means that the underwriters have to use all of their years of experience to decide whether or not to finance somebody.

The Private Equity Firm needed someone to come in, examine the custom-written applications that the company used, and provide an assessment of the current state and the future state. The output of this engagement was a report that detailed all of the things that needed to be fixed in the current versions of the applications, a roadmap for the future state, and recommendations around the development team and development process.

The Litigation Financing company used a low-cost offshore development company to develop the software that managed the cases. One application was written in Visual Basic 6. The main suite was written in VB.NET. The development staff had a bunch of turnover, and the Litigation Finance company was not satisfied with the quality of work.

As an aside, one of the common scenarios where someone might engage my services is to come in and “fix” the work that is done by a low-cost offshored development company. Often times, startups will not have the money to engage high-quality in-house developers, and will farm the initial version of their application out to one of these offshore development shops. Then, when the startup gets some recognition and some investor money, and needs to develop the next stage of their product, they realize that they do not have a good base to work from. This is where I can help out.

I started off by interviewing all of the key stakeholders, and at the same time, dove into the subject of Litigation Finance. I asked very probing questions about the software, their experiences and pain-points with the apps, their interactions with the development company, and where they wanted the future state of the product to be. I also did some analysis of providers of Litigation Financing and Case Management software, and mapped some of the features that I saw into what my client could use in a future-state product.

I then sat down with the source code in order to get a complete understanding of the system. I took the VB.NET code, and using dotPeek from JetBrains, translated the code into C#, which made the code more readable to me. I examined every single module and wrote down a lot of notes about things in the code which could be improved.

Then, I sat down with the senior leadership of both the Private Equity company and the Litigation Finance company, and we had a number of conversations about the future state. There are a lot of technologies that could be incorporated into the product which would make it more powerful and would help my client make better decisions about which injured people to finance. 

At the end, I produced a 15-page report that detailed the fixes for the current state and the migration to the future state, including recommendations around building out an IT organization. I hope to return at a future date to help this client build out the new platform.