Software Development & On-Call Rotation

[noobman]

I'm not sure if this is the right place for general discussion around software development not directly related to code, but here goes:

Recently I've been thinking about the role of software developers in software support. I'm a Java/.NET developer in an IT operations environment, and seeing as how I'm the only developer well-versed in the application's middleware components, I have been placed on the on-call rotation with our only environment guy. With two of us sharing on-call rotation, it works out so that I spend half of the year (26/52 weeks) carrying the pager around. It used to be that we had two environment guys splitting the on-call rotation, but that has since changed.

I'm not sitting at the front line picking up phones to answer RTFM-type questions by any means, but if the application crashes or there's a server issue I'm getting paged to bring it back online. I have no problem acting as L3 support, but to be in an L2 support position restarting services is a little concerning. In the times I have performed L3 support, I feel like I've learned quite a bit about how users interact with our application, and I have been able to take that knowledge with me to help improve application design. Recycling the services because the company refuses to install the application on a box without enough resources is just a nuisance.

Do any of you participate in on-call rotation as part of your lives as software developers? How is your rotation? How do you feel about having to perform support duties?

 

[jwcalla]

I went on a few interviews where they mentioned on-call duties were required and I looked at them like they had an arm growing out of their heads.

IMO if the software is written and QA'd correctly there should be no need for a software dev to be on call. It's just laziness on the part of the company to not do it right in the first place.

Otherwise, if you want me to completely re-arranged my life around your business, you'll have to compensate me accordingly. That's my view at least.

Not that I haven't worked at places where off-hours software support was required (and common), but that's different than being "on-call".

 

[PTNL]

I'm not sure if this is the right place for general discussion around software development not directly related to code, but here goes:

Recently I've been thinking about the role of software developers in software support. I'm a Java/.NET developer in an IT operations environment, and seeing as how I'm the only developer well-versed in the application's middleware components, I have been placed on the on-call rotation with our only environment guy. With two of us sharing on-call rotation, it works out so that I spend half of the year (26/52 weeks) carrying the pager around. It used to be that we had two environment guys splitting the on-call rotation, but that has since changed.

I'm not sitting at the front line picking up phones to answer RTFM-type questions by any means, but if the application crashes or there's a server issue I'm getting paged to bring it back online. I have no problem acting as L3 support, but to be in an L2 support position restarting services is a little concerning. In the times I have performed L3 support, I feel like I've learned quite a bit about how users interact with our application, and I have been able to take that knowledge with me to help improve application design. Recycling the services because the company refuses to install the application on a box without enough resources is just a nuisance.

Do any of you participate in on-call rotation as part of your lives as software developers? How is your rotation? How do you feel about having to perform support duties?

Doing L3 support is totally reasonable, and I would expect that you'd have the occaision escalated/emergency call. Partnering and listening to your frontline or L2 support guys can reveal useful information that can lead to small and significant improvements by your customer.

However, am I understanding this correctly that 50% of the calendar year (not 50% of the calendar weekends) that you are on-call? If so, that kind of random "stop what you're doing, and address this crisis now" can quickly throw any proposed timelines for rolling out new features in your development duties. I see this as your biggest obstacle for your bosses to get new features/fixes/enhancements tested and deployed on a reasonable basis.

Plus... I think part of your question also depends on how management perceives your role - do they regularly work with you on new tasks, or is 95% of your non-support time spent keeping the status-quo.


My primary daily duties are toward software development -- features, fixes, sometimes R&D work with software and hardware, etc. But my week is sometimes speckled with occasional support escalations, plus a weekly meeting with the head of the support team -- very informal, just us chatting about current goings-on and sometimes drilling through an internal bug list site. But I am not in the pager rotation; that sits with about 6-8 individuals which makes up our product support team. Perhaps once every few months I will get an escalated call during the weekend from the on-call support person, but that's just an indirect involvement with their on-call process.

 

[noobman]

However, am I understanding this correctly that 50% of the calendar year (not 50% of the calendar weekends) that you are on-call? If so, that kind of random "stop what you're doing, and address this crisis now" can quickly throw any proposed timelines for rolling out new features in your development duties. I see this as your biggest obstacle for your bosses to get new features/fixes/enhancements tested and deployed on a reasonable basis.

This is absolutely the case. We are on two week rotations, so I spend half my life on-call now. Our L1 support mostly forwards stuff to us, although my boss has talked about trying to offload more of our L2 tasks (IE recycling the services) to the L1 support team, so that might reduce the number of page-outs we get. During on-hours I can usually offload to our environment guy even when I am on-call and he doesn't mind, but off-hours I am on the hook.

Plus... I think part of your question also depends on how management perceives your role - do they regularly work with you on new tasks, or is 95% of your non-support time spent keeping the status-quo.

Our new(ish) manager admittedly comes from a support role, having come up through the organization as part of a true L2 support team. He understand that we do development on our team, but his big mandate is uptime and support structure. Quite frankly it sounds like development is an after-thought for him, and he dismisses me when I try to recommend things like enhancements to our code review process. His idea of "efficient development" is giving something to one person to own the development, full testing, build, and deployment of said code... since his focus is on support. Much of my non-support time is spent working on larger upcoming releases. When I started I did mostly bug fixes, but up until we got our new manager and lost our environment guy I was working on feature development and enhancements.

My primary daily duties are toward software development -- features, fixes, sometimes R&D work with software and hardware, etc. But my week is sometimes speckled with occasional support escalations, plus a weekly meeting with the head of the support team -- very informal, just us chatting about current goings-on and sometimes drilling through an internal bug list site. But I am not in the pager rotation; that sits with about 6-8 individuals which makes up our product support team. Perhaps once every few months I will get an escalated call during the weekend from the on-call support person, but that's just an indirect involvement with their on-call process.

That used to be my role as well, but since losing our other environment guy I've been doing double-duty with all of my tasks plus our laid off environment person's work as well. What happens now is that I get pulled into conversations about ordering/installing SSL certificates, fixing load balancers, or dealing with the Tomcat middleware on top of my development work, and when I am at full capacity they hand off my additional work to some consultants we have on board or pull in a junior developer from another development team in the organization to help out... so things get done wrong and pushed without a code review (since I'm the only person who seems to care about the code review process anymore) which then creates more support issues that take even more time for development work.

It's gotten to the point where I'm seriously considering moving on, which is a shame because I really loved this job and environment.

I went on a few interviews where they mentioned on-call duties were required and I looked at them like they had an arm growing out of their heads.

IMO if the software is written and QA'd correctly there should be no need for a software dev to be on call. It's just laziness on the part of the company to not do it right in the first place.

Otherwise, if you want me to completely re-arranged my life around your business, you'll have to compensate me accordingly. That's my view at least.

Not that I haven't worked at places where off-hours software support was required (and common), but that's different than being "on-call".

I agree with you 100%. When I came on board about a year and a half ago, I was told that the company was looking to expand the team size and add more full-time developers, and that production support / oncall rotation would NOT be part of my regular duties. Of course a year and a half later after missing profit targets, layoffs, and a hiring freeze, I'm in the position I'm in now.

 

[calebb]

We don't have dedicated administrators in the Steam team; instead there are currently ~25 devs, and we rotate carrying the pager for a week at a time. (so, the inverse of your rotation interval ). Pager holder responsibilities for the week includes integrating code changes from the previous week, performing unit/integration/load tests, deploying current bits to the servers, ensuring OS updates have been installed, and most importantly, resolving service-affecting issues as quickly as possible (which could include anything from a DDOS attack to a data center electrical fault).

 

[sharknice]

It depends on what the culture is like where you work, but you may want to tell your boss, or someone higher up that you don't like your current situation and what you would like changed. If they really value you they will try to work something out so you are happier. If you are considering an offer for another position there isn't much to lose trying.

Where I work we have 2 development teams. One team does development of new products and features. The other team is more of a a client support team. They setup our software for new clients and deal with bugs and outages. The main development team is responsible for fixing bugs within the first week or so of a new release that are related to their code.

For us no one person owns code, the whole team owns all of the code and is responsible for it. Even during the initial QA process we will fix each other's bugs.

 

[martin_frg]

It depends on what the culture is like where you work, but you may want to tell your boss, or someone higher up that you don't like your current situation and what you would like changed. If they really value you they will try to work something out so you are happier. If you are considering an offer for another position there isn't much to lose trying.

Where I work we have 2 development teams. One team does development of new products and features. The other team is more of a a client support team. They setup our software for new clients and deal with bugs and outages. The main development team is responsible for fixing bugs within the first week or so of a new release that are related to their code.

For us no one person owns code, the whole team owns all of the code and is responsible for it. Even during the initial QA process we will fix each other's bugs.

I agree with you. A comfortable working environment means a lot for the overall business success.

 

[Dogs]

I went on a few interviews where they mentioned on-call duties were required and I looked at them like they had an arm growing out of their heads.

IMO if the software is written and QA'd correctly there should be no need for a software dev to be on call. It's just laziness on the part of the company to not do it right in the first place.

Otherwise, if you want me to completely re-arranged my life around your business, you'll have to compensate me accordingly. That's my view at least.

Not that I haven't worked at places where off-hours software support was required (and common), but that's different than being "on-call".

Unfortunately that's just not how things work. No amount of best practices and QA an prevent all problems. Even at the most exceptional shops, there are still production problems. And if you're dealing with a 24/7 SLA, you're going to need on call developers as a result of this. Production issues that don't get resolved as quickly as possible can easily cost more money than a developer's day could ever be worth.

What should be true, however, is that developers should rarely get called in shops that are doing things right. Production errors should be infrequent, and when they do occur IT ops should be able to fix most of them. If on call devs are being woken up on a regular basis, then clearly things aren't being done right. Being on call shouldn't be something developers dread.

My point is, on call duties shouldn't throw up warning flags unless there are other warning flags.

Of course, some places may have developers with a more application support focus, which is why other developers don't need to be on call. Some places employ some devs to be support focused and others to be design focused, but these places still have on call... they just don't need all devs to be in the rotation.

 

[CEpeep]

I am the lead systems developer, which also makes me the de facto administrator for those systems. If the servers themselves fail, the hosting team fixes them. If an app fails, I am "on call" and responsible for fixing it, but I designed and helped built it so that hasn't happened yet.

If you have control over and confidence in your work and the people and infrastructure that support it, being "on call" shouldn't really be a big deal. If the pieces are all in place it should be extremely rare that you are "called" upon, and if you are it's because the pieces have gotten seriously munged and you are the one with expertise required to fix it. If you have an SLA with four (or more!) nines reliability as a stipulation your organization simply cannot afford not to have you on call, and neither can the integrity of your work product.

 

[somebrains]

Take control of your support teams.
Document and categorize issues so that you can "train" your L2.
Lead meetings to dole out work, pushing serious BS upwards.
Turn this into resume fodder and get a higher paying job

 

[noobman]

Take control of your support teams.
Document and categorize issues so that you can "train" your L2.
Lead meetings to dole out work, pushing serious BS upwards.
Turn this into resume fodder and get a higher paying job

Thanks!

This is almost exactly what I did and now I"m an ASP.NET web developer. I make 33% more on my base salary and work much less on-call supporting far more resilient applications. The team I left now has tons of automated scripts to handle support with things like basic recycles or cache resets being handled by L2.

 

[4saken]

Dev's absolutely normally have on-call rotations, especially at larger dev shops/companies. Although you wouldnt be responding necessarily to normal operational issues, when there is a large build being deployed, you have to be expected to be able to hop on at the last minute if a show stopping bug that QA missed(which happens lets be honest) on a deploy that has to go out.

 

[somebrains]

I'm glad it worked out for you.
Yards gained after the catch is important.
I have a friend who stepped out in 2008 after 5 years with a visualization vendor.
He's taking the proven process of leading dev and rebooting his career.

It didn't matter that he wasn't up to date, he has the soft skills to show $'s saved in production and warranty work from Day 1.

Thanks!

This is almost exactly what I did and now I"m an ASP.NET web developer. I make 33% more on my base salary and work much less on-call supporting far more resilient applications. The team I left now has tons of automated scripts to handle support with things like basic recycles or cache resets being handled by L2.

 

[v6maro]

I remember doing that, when I was green. Then I learned the fine art of interviewing the company that wanted to hire me to get the truth out of them...I no longer do any support work and just dev, it's nice. Look for a new job, you'll make more money and not have to deal with that bs.

good luck.

 

[Shyne151]

Definitely not on call. I work for a university as a software developer... I do however have a cell phone provided by the university that I am reachable at any time if something pops up.

I'm also more of a devops roll as I maintain several of our Linux web servers.