AT&T support taking a dump

[Mister Natural]

I've known for years that AT&T will drag their feet getting things done when a third party vendor is involved but recently things have become downright hostile. Yesterday I had an outage at a location with business class DSL and the people I had to talk to seemed annoyed and did anything they could to pass the buck to someone else in the chain. Even the tech that arrived on site was useless.

But I have something else that takes the cake. I've never witnessed such hostile behavior until now. Our ISP in the main office is a third party vendor. Back in JULY we initiated a ticket to upgrade our internet from 4 bonded t1's to a 20 Mb connection. This required installing fibre along with the needed hardware.

Since then we've had numerous delays and people intentionally ignoring what it takes to finish getting things done. I'd have to write a book on all the things we've been through but I want to let everyone know just how ignorant AT&T has become.

It took until September for AT&T to get things ready to where our ISP could install their router. When that finally happened it was determined that the patch cable from the MUX to the patch panel was bad. We elected to bypass the patch cable and plugged our cat cable directly into the MUX so we could go ahead and verify the line was up and good to go. We then called AT&T to have them replace the bad cable. Below is a pic of what I left for them.

You can see I clearly tagged the bad cable that needed to be replaced and at the bottom you can see where we bypassed the patch cable and our line is directly into the MUX.



We waited a couple weeks, nothing happened. I checked the dmark last Friday around noon and still nothing. I got a call yesterday and was told an AT&T guy and our ISP tech were going to meet and fix the problem. I went down there at the time given and it turns out someone from AT&T showed up late Friday after I checked. So what did they do?

They removed our line from the mux, snipped it and put a new connector and plugged it back into the patch panel and then removed the tag I put on the bad cable, plugged it back into the mux and left. The ISP tech checked and the connection with the bad cable was still no good.

We cut out the bad cable so AT&T was finally forced to fix the problem. We would have replaced the cable our-self but my boss wanted AT&T to do it and we did not have the right connector for the patch panel anyway.

This behavior is inexcusable and everyone needs to know what AT&T has become.

 

[RocketTech]

I know their phone based support is pretty bad. The tech I've worked with is pretty good. If you can get ahold of their special problem resolution department, you'll have better luck.
Their normal phone support is just trying to funnel you into their pay-for-support company (connect-IT?). I called asking for the next-hop address for my static IPs and no one could tell me, and they thought I was trying to hack their network for even asking. They said I had to pay for support to get that information.
Many field techs are just not trained for anything beyond specific tasks- basically rote learning. What may be readily apparent to you may not even occur to them.

 

[jeffmoss26]

We had our 10 meg fiber upgraded to 20 meg (which required a new circuit and equipment) and overall it was a positive experience.

The tech came onsite, activated a second pair of fibers and installed one of the Ciena boxes you have pictured. A week or so later, we installed the new 1921 router and connected it to the Ciena so the circuit could be tested. Our actual cutover took about 2 minutes, unplugging the cable from our firewall to the previous 1821 router and plugging it into the new 1921.

 

[Mister Natural]

Many field techs are just not trained for anything beyond specific tasks- basically rote learning. What may be readily apparent to you may not even occur to them.

They should be able to at least test a line to determine if it is working.

I haven't mentioned all we've been through. This is why I had pics of the situation before it occurred. It got to the point where I had to document each process to get it moving forward with AT&T. Our ISP had to forward my documentation showing proof of a situation in order to get it moving forward.

 

[RocketTech]

I hear you and know your pain- I'm not contradicting you. From what you've said, you've done everything I would have- document, try to be on site when they are, check their work, etc. If the Service Provider won't cooperate, what can you do? If they are the only provider of the service you need (the case in many locales), you are SOL.
Sometimes getting the regulators involved, or even your elected representatives can help. We were having some issues with CenturyLink and a passing comment to the State Representative returned that the legislature was currently working on some telecom legislation. They were able to drop a hint and the situation was addressed (not well, but it was addressed).
I definitely feel your pain on poor service and not having a lot of choice in addressing the issue.
Take a break and work your way up the chain. In my experience, the area Supervisors aren't typically aware their Techs suck. Try to get ahold of an area Supervisor (State or Regional), usually via Executive Contact or some such. Normal escalation via Phone Support won't get you there. Sometimes Sales department will. If you lodge a formal complaint with a regulating body, it will almost certainly be addressed by an exec or regional supervisor.

My nemesis is CenturyLink- others have no issue with them. I've had very good experiences with Cox, but I've heard horror stories from others. We have a local Cable company called EagleCom that can't seem to troubleshoot themselves out of a paper bag- on their own network. They've sworn straight up and down no one wants or needs 10mbps or higher. They also blame everything on Netflix. Here's a newsflash- when a company creates demand for your product, that's a good thing, not a bad thing.

 

[Mister Natural]

Well the circuit is finally ready to go and I have more than enough to do than pursue it any further. My revenge is posting the experience here for all the world to see.

We also have a backup circuit which is handled by Centurylink. It's not the primary circuit I've been talking about. But I know all about the woes with Centurylink. Any time I had to call them with an outage is was a clusterfuck and run around. I still don't know the circuit ID of that line, even after having a trouble ticket open with them a few times. I've asked and they don't know what it is.

After a couple of years of this I met the local onsite tech for my area and he gave me his cell number and a different number to call when there is a problem. It's made things much better in that regard.

 

[stiltner]

I hate to point out the obvious to you. But most of these jobs where they "send AT&T techs"
its really not. I know because I see the work orders for cutovers and all kinds of other things on
all the platforms.

You'd be surprised what AT&T will farm out to the lowest bidder to come to your premises and
do.

If you want examples, I'll be glad to show you

 

[bds1904]

So you have internet through a CLEC that is forcing AT&T to rent them lines at below cost and you are mad that AT&T is slow to fix a problem that has been bypassed?

If you had AT&T internet service and you had a defective cable from the mux to the patch panel you would get an I&R tech out within 24hrs.

Technically, only the bottom two boxes are AT&T's anyways, the patch panel is actually your ISP's. Whoever put that sticker on there was wrong. If AT&T isn't technically your ISP their responsibility ends at the mux.

I would say you have been calling the wrong company.

The reason why no one from AT&T can help you on the phone is because you aren't their customer, the CLEC is.

Right connector for the patch panel? I call horse shart on that one. It is either RJ45, RJ14 or RJ11. Any network guy should have those if they are worth a damn. Run a premade patch cord.

 

[Mister Natural]

I hate to point out the obvious to you. But most of these jobs where they "send AT&T techs"
its really not. I know because I see the work orders for cutovers and all kinds of other things on
all the platforms.

You'd be surprised what AT&T will farm out to the lowest bidder to come to your premises and
do.

If you want examples, I'll be glad to show you

The in house electrician told me it was AT&T people wearing AT&T logo's on their clothing doing the work. I confirmed this with the guy that showed up after we cut the bad cable and forced him to replace it. He fucked around for an hour, sat in his AT&T truck and finally gave it up and installed a good patch cable because I was there.

So you have internet through a CLEC that is forcing AT&T to rent them lines at below cost and you are mad that AT&T is slow to fix a problem that has been bypassed?

If you had AT&T internet service and you had a defective cable from the mux to the patch panel you would get an I&R tech out within 24hrs.

Technically, only the bottom two boxes are AT&T's anyways, the patch panel is actually your ISP's. Whoever put that sticker on there was wrong. If AT&T isn't technically your ISP their responsibility ends at the mux.

I would say you have been calling the wrong company.

The reason why no one from AT&T can help you on the phone is because you aren't their customer, the CLEC is.

Right connector for the patch panel? I call horse shart on that one. It is either RJ45, RJ14 or RJ11. Any network guy should have those if they are worth a damn. Run a premade patch cord.

You're wrong on all counts. The patch panel was installed by AT&T and is definitely not property of our ISP. It was already there before all this began. The other lines in that patch panel are other offices in the building. The connector on the patch panel is female rj45 where we connect. But it is direct wired into the other end. Nice try on crapping on my post but anyone in this situation knows the connector I am talking about. Like I said I'd have to write a book on all the crap they pulled. As I stated I am used to them delaying projects when third parties are involved,.but this went beyond that. It was deliberate actions to delay the process on all accounts. You think the actions I described in the photo were appropriate? So how long you work for AT&T?

 

[narsbars]

Years ago I was standing in my drive way with a butt set clamped to a broken drop wire. When I called to get it fixed I was told that I had to go to the interface and plug in a phone or they would not dispatch.

 

[Mister Natural]

Years ago I was standing in my drive way with a butt set clamped to a broken drop wire. When I called to get it fixed I was told that I had to go to the interface and plug in a phone or they would not dispatch.

Yep I hear ya.

 

[bds1904]

The patch panel was installed by AT&T and is definitely not property of our ISP.

Just because it was installed by AT&T doesn't mean that it is the point of demarcation. In most cases the equipment is the demarc, not the patch panel.

It was already there before all this began.

So was the sky, does that mean that AT&T owns it?

The connector on the patch panel is female rj45 where we connect. But it is direct wired into the other end.

So you are scared of a regular everyday rj45 punchdown snap in jack?

You think the actions I described in the photo were appropriate?

No, the whole thing was wired incorrectly. If you knew anything about IT wiring you would know that crimping an end onto solid cable is wrong and pitched a fit about it. Should have been female punchdowns at both ends with patch cables. That's the entire reason why the providers responsibility usually end at the equipment.

Perfect example is a non telco managed T1. The telco's responsibility ends at the smart jack.

If you go into any major office building with several ISP's you will see that each ISP has it's own demarc even if the are renting fiber from the local telco. Seeing as you have 1 demarc for a possible many ISP's screams that AT&T's responsibility ends at the equipment.

Bottom line is that you are not an AT&T customer, you are a customer of your CLEC ISP. In the end you should have contacted your ISP who should have come out and either taken care of it or come out and contacted AT&T and coordinated a service call. The reason why you got shitty service is because you called a company who you aren't even doing business with. That very reason is why the phone-ball passing took place, because you didn't have an AT&T account number. In the end if AT&T took care of it for you directly you should be praising their customer service because they took care of something that was requested by a non-customer.

I don't represent AT&T BTW, but I have dealt with your situation before and I also learned the hard way that you call your ISP, not who your ISP leases the cable from. There is a process for a reason.

 

[RocketTech]

Nice job on bds. If anyone thinks incumbents /CLECs are losing money on reselling data lines, you might want to re-think. Sure, they may not be able to charge retail rates set in the 80s based on costs and huge markup, but they are not selling below their costs.
Lets say the CLECs actually are losing money to ISP resellers- They knew the rules when they signed the monopoly agreement, you have to figure they did the math. Offering poor service as a business model is never a good idea.
CLECS complain constantly about the high cost of equipment and infrastructure. Around here, the same copper is on the poles as 20-50 years ago. The same 70s era wiring pillars since the... 70s. If the CLECs were stringing fiber to replace the 50-year old copper on the poles, I'd say they at least have a case. The simple fact is CLECs invest as little as possible in existing infrastructure and replace CO equipment slightly more often, usually to support services they charge an arm and a leg for to support their actual money-losing operations, such as local POTS, long distance, etc.

 

[RocketTech]

bds, I'd stop while I could see daylight. You're wrong, and if you don't know it I dunno if you will listen as to why.
The demarc is wherever the provider says it is. If the provider says it owns all the way up to the patch cable and card in your computer, the demarc is the PCI slot. If a piece of equipment is installed which is labeled 'property of XY&Z, do not touch', that's a pretty strong indication the equipment belongs to XY&Z and you should call them before borking with it.

There is no rule or certification banning male terminations on solid cable. There are situations where it makes more sense to punch-down to a PP or jack, but no requirement- that's why solid RJ45 male terminators are readily available.

Again, if the equipment is labeled as belonging to company XY&Z and is found to be defective, you are suggesting it is OK to modify, replace, fold/spindle/mutilate the clearly labeled equipment? OP clearly stated he worked with the 3rd Party reseller on the issue, so did you not read the OP, or are you making some other case? Why did AT&T dispatch a tech to address the issue if it wasn't their responsibility? Are they incompetent because they don't know their own responsibility, or because they don't know hot to replace a cable?

I do agree with you that your first call is to whomever sold you/provides the service, but we all know sometimes we need to work our way up the chain- be it on a conference call, group meet, whatever. If you are saying you make one call and all your service problems are taken care of, I want to sign up.

 

[bds1904]

OP clearly stated he worked with the 3rd Party reseller on the issue, so did you not read the OP, or are you making some other case?

Really, cause that's not what is in the first post

We then called AT&T to have them replace the bad cable.

I can guarantee his experience would have been better if he called his ISP not AT&T. You can't argue that. If he contacted AT&T, whom he is not a customer of, and they dispatched a tech then AT&T went above and beyond their call of duty.

The story would be different if he called his ISP and his ISP called AT&T and then this happened.

My point is he has less of a leg to stand on because he called someone other than his ISP. It could have been resolved faster and the time-frame reduced if he called who he actually does business through. If his ISP wouldn't call AT&T to get it repaired then he needs to be bashing his ISP also for dropping the ball.

It's like calling the CBS headquarters because your cable is out.

The demarc is wherever the provider says it is. If the provider says it owns all the way up to the patch cable and card in your computer, the demarc is the PCI slot. If a piece of equipment is installed which is labeled 'property of XY&Z, do not touch', that's a pretty strong indication the equipment belongs to XY&Z and you should call them before borking with it.

Actually the FCC defines where the demarc is, and in 99% of situations with leased line broadband equipment, the equipment is the demarc. Just because some idiot tech put a sticker on something doesn't necessarily means it is owned by them. Do you put a sticker with your name on it on a bently at the dealer then drive off with it because it's labeled as yours?

My entire point is that if there is more than 1 providers ports on that patch panel then that patch panel is not the demarc. The FCC clearly states that each provider has to have it's own demarc in an MDU situation. This is so the CLEC can troubleshoot IW without the possibility of interrupting another companies customer.

If you want to get real techinal the demarc in an MDU is always an rj21x for phone and patch panel with an orange cover for data. The mux equipment is supposed to be in a locked cabinet also. To go even farther there are also situations where the MUX equipment is entirely owned by the CLEC. This applies to at the prem and in the CO. In those situations the CLEC has assess to their cage in the CO and maintains the equipment at the prem. The clec'c simply rent the physical line from the telco.

 

[RocketTech]

I really shouldn't feed the troll, but I'm bored.
Maybe I just read the OP differently from you, but I got the impression OP was working with ISP and ATT. I further assumed ATT and was working directly with the OP to resolve the issue.
Just in case I'm wrong though, from the 4th post [Quote:]Our ISP had to forward my documentation showing proof of a situation in order to get it moving forward. [/QUOTE].
But I guess it is easy to assume everyone else in the world is an idiot and doesn't know how to handle problem resolution.
So let me get this straight- you are suggesting that the tech dispatched would have done a better job if the ISP called instead of the OP? Really? Poor service as a business model is a valid response? And we're supposed to take you seriously?

The demarc point is (by rule and in practice) flexible, especially for commercial multi-tenant buildings. The demark seems to be pretty clear here as the equipment is clearly labeled and (I assume) at the MPOE, placing it in clear compliance with FCC rules.

And I really can't believe you are arguing that real (non-sub) ATT techs would have done a better job, that they don't know what equipment they install, they apparently support stuff they shouldn't, they should have installed stuff for an apartment (MDU) in a commercial building- and they didn't install it in a locked cabinet with the 'correct' connections- but the OP is wrong for saying AT&T is doing a poor job??? Do you even understand what side of the argument you are on?

You may be a whiz at wiring up your Mom's basement, but you might want to gain a little more experience before working with commercial customers.

 

[bds1904]

But I guess it is easy to assume everyone else in the world is an idiot and doesn't know how to handle problem resolution.
So let me get this straight- you are suggesting that the tech dispatched would have done a better job if the ISP called instead of the OP? Really? Poor service as a business model is a valid response? And we're supposed to take you seriously?

The demarc point is (by rule and in practice) flexible, especially for commercial multi-tenant buildings. The demark seems to be pretty clear here as the equipment is clearly labeled and (I assume) at the MPOE, placing it in clear compliance with FCC rules.

And I really can't believe you are arguing that real (non-sub) ATT techs would have done a better job, that they don't know what equipment they install, they apparently support stuff they shouldn't, they should have installed stuff for an apartment (MDU) in a commercial building- and they didn't install it in a locked cabinet with the 'correct' connections- but the OP is wrong for saying AT&T is doing a poor job??? Do you even understand what side of the argument you are on?

You may be a whiz at wiring up your Mom's basement, but you might want to gain a little more experience before working with commercial customers.

How about you get a little more experience dealing with commercial telco customers before you claim to know what you are talking about. MDU is defined as any customer unit with multiple tenants that require service under different names. AKA an office building with multiple rented spaces.

As far as the quality of the work, we all know there are good techs and there are bad techs. In this case, the person that put that setup in originally did what I refer to as the "bare-bones" install. While it may be up to expectations of a supervisor, it is not well labeled, organized or secure. I have seen an AT&T Uverse prem tech do better work.

Personally, if I was an at&t tech and I did that install the rack would be enclosed, there would be a 12 port patch panel in the rack that then went to another 12 port patch panel mounted on the wall. The patch panel on the wall would be labeled with a letter/number, each port would be labeled with a letter/number. Patch mounted on the wall would read AT&T only and have an extended orange cover on it. Each CLEC would then have a patch mounted to the right and then vertically that was correctly labeled per port and labeled "to clec x". That crossconnect is the demarc. From there the clec could terminate however they wanted. Depending on how many tenants there are I may take a marker and section off the board for the clec's equipment and label it "clec x only". I'll then run all their patch wires to their "stall", label them individually so they can terminate them. That's how I do installs when dealing with 400+ customers. Each unit would also have a patch mounted in their "unit" stall. Neat and organized. Reason why I go above and beyond? Easier for any tech that ever comes there (my company or not) to fix things. Also means that you don't get someone else messing with your wires. You would be amazed how often that happens.

While there "technically" may not be anything wrong with ends crimped on solid wire, I cut an average of 50-100 off every week and install a jack because they have fatigued, so I guess I have lost my faith. Ask anyone that deals with cat5e/cat6 on a daily basis and they will agree.

 

[Mister Natural]

I've got better things to do. I've made my point.

AMF