C# string substitution (or: Encoding a URI string)

carl67lp · Aug 27, 2004

I need to encode a string before it's sent as a POST request, and will need to also substitute certain characters here and there.

I've looked at the Uri class, but I can't seem to figure out how to use it.

I'll also need to convert "\n" to chr(11) (well, that's the PHP representation--that is, I need to convert newlines into ASCII 11 characters).

Any help?

carl67lp · Aug 27, 2004

Well, I figured out how to use HttpUtility.HtmlEncode() (I didn't realize I had to add a reference...so much yet to learn). So, that takes care of that problem. Now...how do I do a character substitution--the \n to chr(11) bit?

carl67lp · Aug 27, 2004

Another thing... Ampersands are getting converted (say, D&AA goes to D&AA), but it seems like that's a bad thing for the query string. Aside from forbidding the entry of ampersands, what else could I do?

:LJ: · Aug 27, 2004

Sounds like you're trying to send it as a GET request - the querystring doesn't exist in the POST request (in POST, things like "&" won't be a problem). Make sure you specify the HTTP method (sorry, don't know much about C#, so I can't help you on the syntax).

carl67lp · Aug 27, 2004

request.Method is already set to "POST". Lemme post some of the code for those who would like to see it.

BTW: The values array simply has the values of text boxes; parameters is a list of names that the PHP processor expects.

Code:

for (int v=0; v < values.Length; v++)
{
     values[v] = HttpUtility.HtmlEncode(values[v]);
}
			
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "";
for (int i=0; i < values.Length; i++)
{
     postData += parameters[i] + "=" + values[i];
     if (i != values.Length - 1)
     {
          postData += "&";
     }
}
			
byte[] data = encoding.GetBytes(postData);

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType="application/x-www-form-urlencoded";
request.ContentLength = postData.Length;

Stream reqStream = request.GetRequestStream();
// Send the data.
reqStream.Write(data,0,data.Length);
reqStream.Close();

HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
byte[] buf = new byte[8192];
int count = resStream.Read(buf, 0, buf.Length);

// DEBUGGING
MessageBox.Show(this, postData);
// EITHER A TICKET NUMBER OR "ERROR"
MessageBox.Show(this, Encoding.ASCII.GetString(buf, 0, buf.Length));

resStream.Close();

Shadow2531 · Aug 28, 2004

Don't know c#, but in c++, I do it something like this.

Code:

#include <iostream>
#include <string>
#include <bitset>

using namespace std;

int main() {
    const string data = "http://www.google.com";
    string code_it;
    string hexits = "0123456789ABCDEF";
    for (size_t i = 0; i < data.size(); i++) {
        code_it += "%";
        code_it += hexits[bitset<4>(static_cast<int>(data[i]) >> 4).to_ulong()];
        code_it += hexits[bitset<4>(static_cast<int>(data[i])).to_ulong()];
    } 
    cout << code_it << endl;
}

That encodes the string by converting each char to its corresponding hex value.

Maybe you or someone can translate that method to c# and provide any changes needed to fit your needs. Since it only convert 1 char at a time, it shouldn't mess with any html entities etc.

carl67lp · Aug 30, 2004

Shadow2531 said:
Don't know c#, but in c++, I do it something like this. ...

Thanks... Unfortunately I wouldn't know how to convert that all myself, but I can't imagine that C# wouldn't have some sort of built-in class or method for this.

Anyone know of something? Or perhaps I'm going about this all wrong, as :LJ: alluded to?

litesgod · Aug 30, 2004

Shadow2531 said:
Don't know c#, but in c++, I do it something like this

carl67lp said:
Unfortunately I wouldn't know how to convert that all myself

You shouldn't have to convert it. Assuming you are using VS.Net, add a new project to your solution of type managed C++ code. Copy and paste. Load a reference to your new project in the existing project and congratulations, you've just created a .NET library. You may have to play around with Shadow's code a bit to get it to work right (obviously you'll need to change it to a function other than main), but the trouble should be minimal.

carl67lp · Aug 30, 2004

litesgod said:
You shouldn't have to convert it. Assuming you are using VS.Net, add a new project to your solution of type managed C++ code. Copy and paste. Load a reference to your new project in the existing project and congratulations, you've just created a .NET library. You may have to play around with Shadow's code a bit to get it to work right (obviously you'll need to change it to a function other than main), but the trouble should be minimal.

Hadn't thought about that. (Has it become apparent yet that C# is not my first language?

)

However, I'm still thinking there has to be a more elegant--read: included--solution to the problem. PHP and Perl both have string parsing functions; I'll be surprised if C# does not.

Man...I need to go buy the C# Cookbook.

Shadow2531 · Aug 30, 2004

My method of abusing bitset for converting purposes is not ideal, but it does it easily

You can do it as a function, like this

Code:

string encode_hex(const string& data) {
    string code_it;
    string hexits = "0123456789ABCDEF";
    for (size_t i = 0; i < data.size(); i++) {
        code_it += "%";
        code_it += hexits[bitset<4>(static_cast<int>(data[i]) >> 4).to_ulong()];
        code_it += hexits[bitset<4>(static_cast<int>(data[i])).to_ulong()];
    } 
    return code_it;
}

Then you can just encode_hex("blabalbal") or encode_hex(string variable)

[MS] · Aug 30, 2004

Not sure what you are encoding, but to address the original question...

If you are trying to encode the string as if it were going to be passed in an http request, along with replacing all instances of character 0x0A with character 0x0B (sounds like that's what you wanted), then the following code would do that:

Code:

HttpUtility.UrlEncode(str.Replace('\n', (Char)11))

If you want to explicitly escape those characters that wouldn't normally be escaped in a url, then something like the below would work.

Code:

static string HexEscape(string stringData){
  char[] chars = stringData.ToCharArray();
  StringBuilder sb = new StringBuilder(stringData.Length * 3);
  for (int i = 0; i < stringData.Length; i++)
    sb.Append(Uri.HexEscape(chars[i]));
  return sb.ToString();
}

edit: oops, removed extra parameter....

carl67lp · Aug 31, 2004

Thanks, [MS]. That looks about what I need. I'll try it out and report back.

carl67lp · Sep 1, 2004

Well, looks like I'm still in the same boat I was before, but with one small improvement--I can now transmit "D&AA" without it chopping off the "&AA" part.

However, I'm still not getting the line breaks correct. Here's the code I have. The problem is that it's cobbled together from many sources so it's not the best it could be. I'll welcome any suggestions anyone has.

Code:

static string HexEscape(string stringData)
{
	char[] chars = stringData.ToCharArray();
	StringBuilder sb = new StringBuilder(stringData.Length * 3);
	for (int i = 0; i < stringData.Length; i++)
		sb.Append(Uri.HexEscape(chars[i]));
	return sb.ToString();
}

private void createTicketButton_Click(object sender, System.EventArgs e)
{
	string shortDescription = "short";
	string logText = textUpdateInfo.Text;
	string url = "http://www.dev-alumni.wayne.edu/sandbox/ct/postprocessor.php";

	string[] values = new string [8] {
	     textFirstName.Text,
	     textLastName.Text,
	     textUnit.Text,
	     textEmail.Text,
	     textPhone.Text,
	     textAccessId.Text,
	     shortDescription,
	     logText
	     };

	// Re-encode problem characters in the POST query string
	for (int v=0; v < values.Length; v++)
	{
		values[v] = HexEscape(values[v]);
	}
	
	ASCIIEncoding encoding = new ASCIIEncoding();
	string postData = "";
	for (int i=0; i < values.Length; i++)
	{
		postData += parameters[i] + "=" + values[i];
		if (i != values.Length - 1)
		{
			postData += "&";
		}
	}
	
	postData.Replace('\n', (Char)11);
	
	HttpUtility.UrlEncode(postData);

	byte[] data = encoding.GetBytes(postData);

	HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
	request.Method = "POST";
	request.ContentType="application/x-www-form-urlencoded";
	request.ContentLength = postData.Length;

	Stream reqStream = request.GetRequestStream();
	// Send the data.
	reqStream.Write(data,0,data.Length);
	reqStream.Close();

	HttpWebResponse response = (HttpWebResponse)request.GetResponse();
	Stream resStream = response.GetResponseStream();
	byte[] buf = new byte[8192];
	int count = resStream.Read(buf, 0, buf.Length);

	// DEBUGGING
	MessageBox.Show(this, postData);
	// EITHER A TICKET NUMBER OR "ERROR"
	MessageBox.Show(this, Encoding.ASCII.GetString(buf, 0, buf.Length));

	resStream.Close();
}

[MS] · Sep 1, 2004

carl67lp said:
Well, looks like I'm still in the same boat I was before, but with one small improvement--I can now transmit "D&AA" without it chopping off the "&AA" part.

However, I'm still not getting the line breaks correct. Here's the code I have. The problem is that it's cobbled together from many sources so it's not the best it could be. I'll welcome any suggestions anyone has.

Ah, given that code, I have a better idea of what you are trying to do. The HttpUtility.UrlEncode() is indeed the function you want to use. Note that in your current code you use the function, but don't use the string that is returned from it. There's a couple other changes I'd make in order to simplify things...

Take a look at the following code...

Code:

StringBuilder sb = new StringBuilder();
for (int i = 0; i < values.Length; i++)
{
  sb.Append(HttpUtility.UrlEncode(parameters[i]));
  sb.Append('=');
  sb.Append(HttpUtility.UrlEncode(values[i]));
  if (i != values.Length - 1){
    sb.Append('&');
  }
}
byte[] dataBytes = Encoding.ASCII.GetBytes(sb.ToString())

* Switched to string builder because for doing a large number of string appends, it'll be much faster.
* UrlEncodes both the field name and value, per the html spec

I don't think you need to manually encode anything or remove the newlines from the strings. The UrlEncode function will take care of all that. The main issue you had was that the output of http encoding wasn't being used, and even if that was fixed, the logic was out or order (you would have also caused '&' and '=' characters to be encoded. Because the http encode wasn't done, it probably seemed as though things needed to be escaped manualy...

Here is full code if you want to take a quick look...
It has an extra function at the end to show the contents of the byte array.

Code:

#region Using directives
using System;
using System.Collections.Generic;
using System.Text;
using System.Web;
using System.Net;
using System.Text.RegularExpressions;
#endregion

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            createTicketButton_Click();
            Console.ReadLine();
        }

        private static string[] parameters = {
            "FirstName", "Last Name", "Unit", "Email", "Phone", "AccessID", "Description", "log"
        };

        private static void createTicketButton_Click()
        {
            string url = "http://foo.com";

            string[] values = new string[8] {
	     "Richie", //textFirstName.Text,
	     "Meyer", //textLastName.Text,
	     "Trinity", //textUnit.Text,
	     "richiem@microsoft.gotspam.com", //textEmail.Text,
	     "(555) 555-5555", //textPhone.Text,
	     "12345", //textAccessId.Text,
	     "A luggage combo", //shortDescription,
	     "logtext?  we'll just put some special " + Environment.NewLine + " characters here... !@#$%^&*()_+={}:<>?" //logText
	     };

            

            StringBuilder sb = new StringBuilder();
            for (int i = 0; i < values.Length; i++)
            {
                sb.Append(HttpUtility.UrlEncode(parameters[i]));
                sb.Append('=');
                sb.Append(HttpUtility.UrlEncode(values[i]));
                if (i != values.Length - 1){
                    sb.Append('&');
                }
            }

            byte[] dataBytes = Encoding.ASCII.GetBytes(sb.ToString());

            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
            request.Method = "POST";
            request.ContentType = "application/x-www-form-urlencoded";
            request.ContentLength = dataBytes.Length;
            // ...
            // ...
            // ...

            DecodeAndShow(dataBytes);
        }


       
        
        private static void DecodeAndShow(byte[] dataBytes){
            string dataString = Encoding.ASCII.GetString(dataBytes);
            string[] fields = dataString.Split('&');
            for (int i = 0; i < fields.Length; i++){
                int equalLoc = fields[i].IndexOf('=');
                if (equalLoc >= 0){
                    string name = HttpUtility.UrlDecode(fields[i].Substring(0, equalLoc));
                    string value = HttpUtility.UrlDecode(fields[i].Substring(equalLoc + 1));
                    Console.WriteLine(name + ": " + value);
                }
            }
        }
    }
}

carl67lp · Sep 3, 2004

[MS] said:
...
I don't think you need to manually encode anything or remove the newlines from the strings. The UrlEncode function will take care of all that. The main issue you had was that the output of http encoding wasn't being used, and even if that was fixed, the logic was out or order (you would have also caused '&' and '=' characters to be encoded. Because the http encode wasn't done, it probably seemed as though things needed to be escaped manualy...

Thanks [MS]...I'll have to look over the code more closely, but before I do, I need to ask: Is it removing newlines and converting to char(11)? It's not that I needed to do so for Web transmission, but I need to do so for the campus server--it takes a newline to mean "end of input" and a char(11) is needed instead.

Sorry if it's self-evident.

Also, I've converted the whole thing to use a Web service--but even that's not working right now. So I may still go with whatever we have here.

[MS] · Sep 3, 2004

carl67lp said:
Thanks [MS]...I'll have to look over the code more closely, but before I do, I need to ask: Is it removing newlines and converting to char(11)? It's not that I needed to do so for Web transmission, but I need to do so for the campus server--it takes a newline to mean "end of input" and a char(11) is needed instead.

Sorry if it's self-evident.

Also, I've converted the whole thing to use a Web service--but even that's not working right now. So I may still go with whatever we have here.

If you need to convert a newline to something different on the server, that should really happen on the server.

Per the html spec that I linked to, newlines are supposed to be transmitted as a Carriage Return (CR, 0x0d) followed by a Line Feed (0x0a). The UrlEncode() does exactly that, converting a newline to the encoded string %0d%0a. If you really want to, you can go against the spec and do that character replacement on the client, but Id advise against that, itll only hurt you in the long run.

Id think there would be some form of UrlDecode() function that you could use on the server side, but my knowledge about the PHP APIs is pretty limited. If there is such a function, it should understand the url encoded newline and convert that to whatever the native newline representation is for that system. If there is no such function and you have to unescape everything manually, you can just do a substring replace or do the replace as part of the decoding.

edit: ah... guess you may not be able to change the server behavior...

If you were asking if it encoded the newlines, yes it does. Each newline is encoded as "%0d%0a", per the html spec.

As for the server behavior, do you mean that the server doesn't process the encoded string after a newline character? If that's the case, no problem, all the newlines will be encoded; there won't be any character 0x0d's in the string.

If you instead meant that the newlines in the actual field values is bad, then you'd be forces to remove those with a character raplace on the client.

carl67lp · Sep 7, 2004

[MS] said:
If you need to convert a newline to something different on the server, that should really happen on the server. ...

You know, I really hadn't even thought of doing it that way. That really would be the simplest, of course--since the POST data should transmit everything just fine, newlines and all, I can use my PHP script to strip out those newlines and replace them with char(11) as the campus server requires.

I'm still wondering, though--would it be simplest to do a Web service, as I'm working on now, or just go with the POST data and whatnot like you posted?

carl67lp · Sep 7, 2004

I've gone ahead and implemented [MS]'s suggested code, modified a bit for my purposes, and it works splendidly. Looks like I had most of it already, but that's sort of like saying I had flour, eggs, and sugar and thus had most of a chocolate chip cookie.

I need to work on saving of data fields between sessions (that is, the user data--I don't want people to have to re-enter the same stuff every single time). That's a whole other task, though.

Incidentally, it was the right idea to do the newlines to chr(11) on the server.

Thanks again, [MS]!

[MS] · Sep 8, 2004

The ease of using a web service all depends on the situation.

If the schools server exposes a web service and you simply want to proxy the request through your system, its extremely simple. The following code would be all thats needed (the system I happen to be at has a VB-only install of VS, though I prefer C#).

Code:

    <WebMethod()> _
    Public Function MakeTicketProxy(ByVal FirstName As String, ByVal LastName As String, ByVal Unit As String) As Integer
        Dim SchoolTicketService As New CSsrv01.TicketService()
        Return SchoolTicketService.MakeTicket(FirstName, LastName, Unit)
    End Function

Of course, you would have more parameters, but its obviously trivial code. Because of the WebMethod attribute on the method, VS and the compilers will take care of generating everything needed to expose the MakeTicketProxy call available as a web service. The first line in the function creates an instance of the schools service (the TicketService class is automatically generated when you add a reference to the schools web service, and the CSsrv01 is just the server name in my example). The second line calls the real service exposed by the school, passing the values sent from the client, and returns and integer which would be the Tickets ID in my example.

The code demonstrates both exposing a web service, and consuming one, its similarly easy to use the service in the code above from an end client application (or a web page).

In the event that the school doesnt support web services for what you need

You could expose a web service from your system, and have it include the logic to translate the request into an HTTP Post of the form data to make it work. In this case, youd move the encoding code you have now into the web service and then simply use the web service from any client apps. This gives you the advantage of making client apps simpler and not having to include the encoding source code in each one. It has the disadvantage that its not as easily directly consumed by down level projects that use POSTs directly. It all depends on what clients you think will consume the proxied service.

The other alternative is to simply expose, as the proxy, something that takes POST data and POSTs that to the schools server. This has the advantage of making things easier for projects that cant consume web services, and the disadvantages of making client apps more complicated and having duplicate code in the different client apps.

carl67lp · Sep 9, 2004

The campus server doesn't have any sort of Web services enabled. All communication to that machine is done via sockets. I have PHP code to do it, and I attempted to turn it into a Web service, but it didn't fly. The WSDL came out with incorrect return types (all strings) even when I specified boolean and so forth. Also, I was a bit confused by some of the dispatch map stuff in PHP (of course, this is all separate from C#).

Maybe I'll make a new thread for all of this, if I think that Web services would still make things easier (somehow, that is the way I see it...).

Thanks again for your help!

C# string substitution (or: Encoding a URI string)

Supreme [H]ardness

Supreme [H]ardness

Supreme [H]ardness

[H]ard|Gawd

Supreme [H]ardness

[H]ard|Gawd

Supreme [H]ardness

n00b

Supreme [H]ardness

[H]ard|Gawd

[H]ard|Gawd

Supreme [H]ardness

Supreme [H]ardness

[H]ard|Gawd

Supreme [H]ardness

[H]ard|Gawd

Supreme [H]ardness

Supreme [H]ardness

[H]ard|Gawd

Supreme [H]ardness