Does such a C# class exist?

[Red Squirrel]

This is sorta in reference to this thread but since it's a separate request figured I'd start a new thread.

I am about to code myself a C# class that will basically be a string class, but will be UTF8, so each char will be one byte, and I will be able to assign values to each byte of a string, add bytes on top etc... pretty much like the C++ string class. I will probably add some more functionality such as a function like WriteInt32 which will write an int into 4 bytes of space in it's binary value.

Before I get into this, does such a class already exist?

If not, and I go ahead with mine, what would be the most efficient way of storing bytes of data in a dynamic fashion? I'm thinking of using a <List> of bytes, but wondering if there's a better way. a byte[] array is no good as I need to predefine how big it is. I suppose I could do that, then just grow it as needed by redeclaring it in buffers, but that will probably be bad in performance.

 

[mikeblas]

In UTF8, sometimes a character is more than one byte.

I thought I already answered your question in that other thread; there's no string class that does what you're asking for. You can use a combination of string and the encoding objects to convert back and forth. If you're worried about doing that too much, then you need to find a container that caches the conversion, or fix your code so you're not using UTF8.

As an aside, UTF8 sucks; you probably shouldn't be using it, especially given the concerns you've expressed about efficiency in the other thread.

 

[Red Squirrel]

UTF8 is needed if I want to work at the byte level. UTF16 does some weird stuff. I need to be able to have control over every byte as the data at the end is going over a network, and when dealing with network streams, it's bytes. Along with jsut regular text, I also will be putting ints in a "packed" format.

Ex: If I want to write a 16-bit number 24930 then it's value would be "ab" or 61 62 in hex.

 

[mikeblas]

If you want to work at the byte level, you should use "unsigned int 8". On Intel machines, 24930 is actually 0x6261, not 0x6162, because Intel machines are little-endian.

Since you're not able to articulate what it is you're doing, perhaps you might offer some example code that shows more specifically what your goal is.

 

[Whatsisname]

Based on your other thread, I recommend looking at System.Text.ASCIIEncoding, including System.Text.ASCIIEncoding.UTF8 and play around with the stuff in there.

 

[Red Squirrel]

If you want to work at the byte level, you should use "unsigned int 8". On Intel machines, 24930 is actually 0x6261, not 0x6162, because Intel machines are little-endian.

Since you're not able to articulate what it is you're doing, perhaps you might offer some example code that shows more specifically what your goal is.

Basically, here's a class I wrote in C++:

/*
======================================================
rslib component:  bitstream implimentors
======================================================
*/


namespace rslib
{

    BitStream::BitStream()
    {
    }
	
	BitStream::~BitStream()
	{
	}
    


	bool BitStream::Lock(bool block=true)
	{
		#ifdef rslib_pthread
		if(block)mutex.Lock();
		else return mutex.TryLock();
		#endif

		return true;
	}

	
	void BitStream::Unlock()
	{
		#ifdef rslib_pthread
		mutex.UnLock();
		#endif
	}
	


    string BitStream::ToStr()
    {
    	string temp;    
    
    	bool byte[8] = {0,0,0,0,0,0,0,0}; //used to store 8 bit chunks
    
    	unsigned int ibyte; //used to construct a byte as a number    
    
        //Note: this assumes a bitstream with multiples of 8. Any extra bits are dropped.    
    
    	for(int i=0;(i+8)<=stream.size();i+=8)
    	{
            
            ibyte=0;
    		    		
    		//get one byte:
    		byte[0]=stream[i];        
    		byte[1]=stream[i+1];   
    		byte[2]=stream[i+2];   
    		byte[3]=stream[i+3];   
    		byte[4]=stream[i+4];   
    		byte[5]=stream[i+5];   
    		byte[6]=stream[i+6];   
    		byte[7]=stream[i+7];   
    		
    		//do stuff with bits and stuff:
			if(byte[0])ibyte |=128;	
    		if(byte[1])ibyte |=64;	
    		if(byte[2])ibyte |=32;	
    		if(byte[3])ibyte |=16;
    		if(byte[4])ibyte |=8;	
    		if(byte[5])ibyte |=4;	
    		if(byte[6])ibyte |=2;	
    		if(byte[7])ibyte |=1;	
    
    		temp.append(1,(unsigned char)ibyte);
    	}
    
    	return temp;
    }
	
	
	string BitStream::ToHex(int groupnum,bool showx)
	{
        stringstream out;
		unsigned int ibyte;
		bool byte[8] = {0,0,0,0,0,0,0,0}; //used to store 8 bit chunks	
		int ii=groupnum;
		
		if(groupnum<1 && showx)out<<"0x";
		
		for(int i=0;(i+8)<=stream.size();i+=8)
		{			
		
			if(ii>=groupnum && groupnum>0)
			{					
				ii=0;
				if(i>=8)out<<" ";
				if(showx)out<<"0x";	
			}
			ii++;
			

			ibyte=0;
						
			//get one byte
			byte[0]=stream[i];        
			byte[1]=stream[i+1];   
			byte[2]=stream[i+2];   
			byte[3]=stream[i+3];   
			byte[4]=stream[i+4];   
			byte[5]=stream[i+5];   
			byte[6]=stream[i+6];   
			byte[7]=stream[i+7];   
			
			//This is backwards and is intentional
			if(byte[0])ibyte |=128;	
			if(byte[1])ibyte |=64;	
			if(byte[2])ibyte |=32;	
			if(byte[3])ibyte |=16;
			if(byte[4])ibyte |=8;	
			if(byte[5])ibyte |=4;	
			if(byte[6])ibyte |=2;	
			if(byte[7])ibyte |=1;

			if(ibyte<0xf)out<<"0";
			out<<hex<<uppercase<<ibyte;		

		}
		
		
        out << hex <<uppercase;
        return out.str();
	}
	
	

	
    
    string BitStream::ToBitStr(bool bytespaces=false)
    {
        string ret;
        int bytepos=0;
        for(int i=0;i<stream.size();i++)
        {
            ret+= (ReadBit(i) ? "1" : "0");
            
            if(bytespaces)
            {
                bytepos++;
                if(bytepos>=8)
                {
                    ret+=" ";
                    bytepos=0;
                }
            }
        }

        return ret;
    }
    

    
    
    string BitStream::Hexdump()
    {
		return HexDump(ToStr());
    }
    
    void BitStream::DisplayInfo()
    {
        cout<<endl<<"Stream size: "<<stream.size()<<"/"<<stream.max_size();
        cout<<endl<<"Max bits per variable: "<<MaxBits()<<endl;
    }
    
    void BitStream::Display()
    {
        int pos8=0;
        
    
    	for(int i=0;i<stream.size();i++)
    	{
        	cout<<stream[i];
        	if(pos8==7)cout<<" ";
        	
        	if(pos8>=7)pos8=0;
        	else pos8++;
    	}
       cout<<flush;
    }
    
    void BitStream::Clear()
    {
    	stream.clear();
    }
    
    BitStream::bsint BitStream::Size()
    {
       return stream.size();
    }
    
    bool BitStream::Empty()
    {
        return (Size()<=0);
    }
    
    int BitStream::MaxSize()
    {
       return stream.max_size();
    }
    
    void BitStream::DeleteBits(int pos,int todel)
    {
		if(pos<0)pos=0;
        else if(pos>=stream.size())return;
        else if((pos+todel)>=stream.size())stream.erase(stream.begin()+pos,stream.end());
        else stream.erase(stream.begin()+pos,stream.begin()+pos+todel);        
    }
	

	
	
	void BitStream::ReverseBits(int pos,int amt)
	{
		if(pos<0)pos=0;
		if((pos+amt)>Size())return;
		
		bool * tmp = new bool[amt];
		
		//write out stream to temp array:
		
		for(int i=0;i<amt;i++)
		{
			tmp[i]=	stream[pos+i];
		}
		
		//re-write it back to stream in reverse:
		
		int i2=amt-1;
		
		for(int i=0;i<amt;i++)
		{
			stream[pos+i]=tmp[(i2)];	
			i2--;
		}
	
		delete[] tmp;
	}
	
	//similar logic as reversebits, but does it in byte chunks	
	void BitStream::ReverseBytes(int pos,int amt)
	{
		if(pos<0)pos=0;
		if(((pos)+(amt*8))>Size())return;	
		
		
		
		BitStream::bsint * tmp = new BitStream::bsint[amt];
		
		//write out stream to temp array:
		
		for(int i=0;i<amt;i++)
		{
			tmp[i]=	ReadInt8(pos+(i*8));
		}
		
		//first clear what we got, then just re-add:
		
		DeleteBits(pos,amt*8);
		
		
		//re-write it back to stream in reverse:
		
		int i2=amt-1;
		
		for(int i=0;i<amt;i++)
		{			
			WriteInt8(tmp[i2],pos+(i*8));	
			i2--;
		}
	
		delete[] tmp;	
	
	}
			
        
    
    //writers:
    //--------
    

    
    
    void BitStream::WriteBit(bool tw,int pos=-1)
    {
        if(pos>=stream.size())pos=stream.size();
        else if(pos==-1)pos=stream.size();
        
		if(pos==stream.size())stream.push_back(tw);
        else stream.insert(stream.begin()+pos,tw); 
    }
    
    
    
    //writes a int based on specified lenght (bits)
    void BitStream::WriteVarInt(BitStream::bsint tw,int bits,int pos=-1)
    {
        if(pos>stream.size())pos=stream.size();
        else if(pos==-1)pos=stream.size();
        
    	if(bits>MaxBits())bits=MaxBits();
    	if(bits<1)bits=1;
    
    	BitStream::bsint orer=1;	
    	for(int i=1;i<bits;i++)
    	{
    	   orer*=2;
    	}
    	
    	//make sure tw is not past its capacity
    	tw = (tw>((orer*2)-1)) ? ((orer*2)-1) : tw;
    	
    	for(int i=0;i<bits;i++)
    	{
        	WriteBit((((orer & tw)==orer) ? true : false),pos+i); 
        	orer/=2;
    	}
    }
    
    
    
    void BitStream::WriteInt8(BitStream::bsint tw,int pos=-1)
    {
       WriteVarInt(tw,8,pos);
    }
    
    void BitStream::WriteInt16(BitStream::bsint tw,int pos=-1)
    {
       WriteVarInt(tw,16,pos);
    }
    
    void BitStream::WriteInt24(BitStream::bsint tw,int pos=-1)
    {
       WriteVarInt(tw,24,pos);
    }
    
    void BitStream::WriteInt32(BitStream::bsint tw,int pos=-1)
    {
       WriteVarInt(tw,32,pos);
    }
    
    
    
    void BitStream::WriteString(string tw,int pos=-1)
    {
        if(pos>stream.size())pos=stream.size();
        else if(pos==-1)pos=stream.size();
				
    	for(int i=0;i<tw.size();i++)
    	{
    	   WriteInt8(((unsigned int)(unsigned char)tw[i]),pos==-1 ? pos : pos+(i*8));
    	}  
    
    }
    
    void BitStream::WriteBitStream(BitStream tw,int pos=-1)
    {
        if(pos>stream.size())pos=stream.size();
        
        for(int i=0;i<tw.Size();i++)WriteBit(tw.ReadBit(i),pos==-1 ? pos : pos+i);
    }
    
    //obsolete: kept for backwards compatibility:
	void BitStream::WriteBitstream(BitStream tw,int pos=-1)
	{
		WriteBitStream(tw,pos);
	}
    
    
    
    
    //readers
    //--------
    
    
    bool BitStream::ReadBit(int pos)
    {
        if(stream.size()<=0)return false;
        
    	return stream[pos];
    }
    
    

    BitStream::bsint BitStream::ReadVarInt(int pos,int bits)
    {
        if(stream.size()<=0)return 0;
        if(pos>=stream.size())return 0;
        if((pos+bits)>=stream.size())bits=stream.size()-pos;
        
    	if(bits>MaxBits())bits=MaxBits();
    	if(bits<1)return 0;
    	
    	BitStream::bsint tmp=0;
    	
    	BitStream::bsint orer=1;	
    	for(int i=1;i<bits;i++)
    	{
    	  orer*=2;
    	}
    	
    	for(int i=0;i<bits;i++)
    	{
    		if(stream[pos+i])tmp |=orer;
    		orer/=2;
    	}
    
    	return tmp;
    }
    
    BitStream::bsint BitStream::ReadInt8(int pos)
    {
       return ReadVarInt(pos,8);
    }
    
    BitStream::bsint BitStream::ReadInt16(int pos)
    {
       return ReadVarInt(pos,16);
    }
    
    BitStream::bsint BitStream::ReadInt24(int pos)
    {
      return ReadVarInt(pos,24);
    }
    
    BitStream::bsint BitStream::ReadInt32(int pos)
    {
       return ReadVarInt(pos,32);
    }
    
    
    string BitStream::ReadString(int pos,int chars)
    {
        if(stream.size()<=0)return "";
    	string toreturn="";
    
    	for(int i=0;i<chars;i++)
    	{
        	int bitpos=pos+(i*8);
        	if(bitpos>=Size())break;
        
        	toreturn.append(1,(unsigned int)(unsigned char)ReadInt8(bitpos));
    	}
    
    	return toreturn;
    }
    
    
    BitStream BitStream::ReadBitStream(int pos,int bits)
    {
        BitStream toreturn;
        if(stream.size()<=0)return toreturn;
    
    	for(int i=0;i<bits;i++)
    	{
        	int bitpos=pos+i;
        	if(bitpos>=Size())break;
        
        	toreturn.WriteBit(ReadBit(bitpos));
    	}
    
    	return toreturn;
    }    
	
	 //obsolete: kept for backwards compatibility
	 BitStream BitStream::ReadBitstream(int pos,int bits)
	 {
		return ReadBitStream(pos,bits);
	 }
    
    
    
    //poppers
    //--------
    

    bool BitStream::PopBit_s()
    {
        if(stream.size()<=0)return false;
        bool pop=stream[0];
        stream.erase(stream.begin());
        
        return pop;
    }
    
    bool BitStream::PopBit_e()
    {
        if(stream.size()<=0)return false;
        bool pop = stream[stream.size()-1];
        stream.erase(stream.end());
        
        return pop;            
    }


    BitStream::bsint BitStream::PopIntVar_s(int bits)
    {
        if(bits>MaxBits())bits=MaxBits();
        
        if(bits>stream.size())bits=stream.size();
        if(bits<=0)return 0;        
        
        BitStream::bsint ret = ReadVarInt(0,bits);
        
        DeleteBits(0,bits);
        
        return ret;
    }
    
    BitStream::bsint BitStream::PopIntVar_e(int bits)
    {
        if(bits>MaxBits())bits=MaxBits();
        
        if(bits>stream.size())bits=stream.size();
        if(bits<=0)return 0;
        
        BitStream::bsint ret = ReadVarInt(stream.size()-bits,bits);
        
        DeleteBits(stream.size()-bits,bits);
        
        return ret;
    }


    BitStream::bsint BitStream::PopInt8_s()
    {
        return PopIntVar_s(8);
    }
    
    BitStream::bsint BitStream::PopInt8_e()
    {
        return PopIntVar_e(8);
    }
    
    BitStream::bsint BitStream::PopInt16_s()
    {
        return PopIntVar_s(16);
    }
    
    BitStream::bsint BitStream::PopInt16_e()
    {
        return PopIntVar_e(16);
    }
    
    BitStream::bsint BitStream::PopInt24_s()
    {
        return PopIntVar_s(24);
    }
    
    BitStream::bsint BitStream::PopInt24_e()
    {
        return PopIntVar_e(24);
    }
    
    BitStream::bsint BitStream::PopInt32_s()
    {
        return PopIntVar_s(32);
    }
    
    BitStream::bsint BitStream::PopInt32_e()
    {
        return PopIntVar_e(32);
    }
    
    
    string BitStream::PopString_s(int chars)
    {
        if((chars*8)>stream.size())chars=(stream.size()/8);
        if((chars*8)<=0)return "";

        string toreturn = ReadString(0,chars);
        
        DeleteBits(0,toreturn.size()*8);
        return toreturn;
    }
    
    
    string BitStream::PopString_e(int chars)
    {        
        if((chars*8)>stream.size())chars=(stream.size()/8);
        if((chars*8)<=0)return "";
        
        string toreturn = ReadString(stream.size()-(chars*8),chars);
        
        
        DeleteBits(stream.size()-(toreturn.size()*8),toreturn.size()*8);
        
        return toreturn;
    }
    
    
    BitStream BitStream::PopBitstream_s(int bits)
    {
        BitStream ret;        
        if(bits>stream.size())bits=stream.size();
        if(bits<=0)return ret;        
        
        ret.WriteBitstream(ReadBitstream(0,bits)); 
        DeleteBits(0,bits);
        
        return ret;
    }
    
    
    BitStream BitStream::PopBitstream_e(int bits)
    {        
        BitStream ret;  
        if(bits>stream.size())bits=stream.size();
        if(bits<=0)return ret;
        
        ret.WriteBitstream(ReadBitstream(stream.size()-bits,bits));        
        DeleteBits(stream.size()-bits,bits);
        
        return ret;        
    } 
    
    
    //pushers
    //-------
    
    void BitStream::PushBit_s(bool tw)
    {
        WriteBit(tw,0);
    }
    
    void BitStream::PushBit_e(bool tw)
    {
        WriteBit(tw,stream.size());
    }
    
    void BitStream::PushIntVar_s(BitStream::bsint tw,int bits)
    {
        WriteVarInt(tw,bits,0);
    }
    
    void BitStream::PushIntVar_e(BitStream::bsint tw,int bits)
    {
        WriteVarInt(tw,bits,stream.size());
    }
    
    void BitStream::PushInt8_s(BitStream::bsint  tw)
    {
        PushIntVar_s(tw,8);
    }
    
    void BitStream::PushInt8_e(BitStream::bsint  tw)
    {
        PushIntVar_e(tw,8);
    }
    
    void BitStream::PushInt16_s(BitStream::bsint  tw)
    {
        PushIntVar_s(tw,16);
    }
    
    void BitStream::PushInt16_e(BitStream::bsint  tw)
    {
        PushIntVar_e(tw,16);
    }
    
    void BitStream::PushInt24_s(BitStream::bsint  tw)
    {
        PushIntVar_s(tw,24);
    }
    
    void BitStream::PushInt24_e(BitStream::bsint  tw)
    {
        PushIntVar_e(tw,24);
    }
    
    void BitStream::PushInt32_s(BitStream::bsint  tw)
    {
        PushIntVar_s(tw,32);
    }
    
    void BitStream::PushInt32_e(BitStream::bsint  tw)
    {
        PushIntVar_e(tw,32);
    }
    
	void BitStream::PushString_s(string tw)
	{
		WriteString(tw,0);
	}
	
	void BitStream::PushString_e(string tw)
	{
		WriteString(tw);
	}
	
	void BitStream::PushBitstream_s(BitStream tw)
	{
		WriteBitstream(tw,0);
	}
	
	void BitStream::PushBitstream_e(BitStream tw)
	{
		WriteBitstream(tw);
	}
    	
    
    
    //overloaders
    //-----------

    
    BitStream& BitStream::operator=(BitStream& tw)
    {
    	if (this == &tw) return *this;    //make sure we're not assigning ourselves to ourselves
    
    	Clear();
    	WriteBitstream(tw);
    	return *this;
    }
    
    BitStream& BitStream::operator=(string& tw)
    {    
    	Clear();
    	WriteString(tw);
    	return *this;
    }  
    
    BitStream& BitStream::operator=(BitStream::bsint tw)
    {    
    	Clear();
    	
    	WriteBitstream(Int2BinBS(tw));
    	return *this;
    } 
	
    
    BitStream& BitStream::operator+=(BitStream& tw)
    {
    	WriteBitstream(tw);
    	return *this;
    }
        
    BitStream& BitStream::operator+=(string tw)
    {
    	WriteString(tw);
    	return *this;
    }
    
    BitStream& BitStream::operator+=(BitStream::bsint tw)
    {
    	WriteBitstream(Int2BinBS(tw));
    	return *this;
    }
    
    
    bool BitStream::operator==( BitStream & tw)
    {
        return (ToBitStr()==tw.ToBitStr());
    }
    
    bool BitStream::operator==( string tw)
    {
        return (ToStr()==tw);
    }    
    
    
    
    
    //statics/utility
    //---------------
    
    int BitStream::MaxBits()
    { 
        return MaxBitsOf(ULONG_MAX);
    }
    
    int BitStream::MaxBitsOf(BitStream::bsint a)
    { 
        return (int)ceil(log(a)/log(2));
    }    
    
    deque<bool> BitStream::Int2Bin(BitStream::bsint tc)
    {
    	BitStream::bsint ander=1;
    	
    	deque<bool> ret;
    	
    	while(ander<=tc && ander>0)
    	{
            if(tc & ander) ret.push_front(true);            
            else ret.push_front(false);   
            ander*=2;
        }
        
        if(ret.size()<=0)ret.push_front(false);
               
        return ret;
    }
    
    
    string BitStream::Int2BinStr(BitStream::bsint tc)
    {
        deque<bool> tmp = Int2Bin(tc);
        
        string ret="";
        
        for(int i=0;i<tmp.size();i++) ret+= tmp[i]? "1" : "0";
        
        return ret;
    } 
    
    BitStream BitStream::Int2BinBS(BitStream::bsint tc)
    {
        deque<bool> tmp = Int2Bin(tc);
        
        BitStream ret;
        
        for(int i=0;i<tmp.size();i++) ret.WriteBit(tmp[i]);
        
        return ret;
    }    
    
    string BitStream::Int2Str(BitStream::bsint tc)
    {
        stringstream out;
        out << tc;
        return out.str();
    }   
	
    string BitStream::Int2HexStr(BitStream::bsint tc,bool dontshowx=false)
    {
        stringstream out;
		
		if(!dontshowx)out<<"0x";		
        out << hex <<uppercase<<tc;
        return out.str();
    }   	
	
	string BitStream::Str2HexStr(string tc,int groupnum=1,bool showx=true)
	{
		stringstream out;

		int ii=groupnum;
		
		if(groupnum<1 && showx)out<<"0x";
		
		for(int i=0;i<tc.size();i++)
		{			
		
			if(ii>=groupnum && groupnum>0)
			{					
				ii=0;
				if(i>0)out<<" ";
				if(showx)out<<"0x";	
			}
			ii++;
			
			out<<hex<<uppercase<<(unsigned int)(unsigned char)tc[i];			
		}
		

       return out.str();
	}
	
    string BitStream::Double2Str(long double tc)
	{
        stringstream out;
        out << tc;
        return out.str();
	}

	BitStream::bsint BitStream::Str2Int(string tc)
	{
		return atol(tc.c_str());
	}	
}

This class allows me to push various data in a single container, which can then be converted to a C++ string (which is bytes) and used to send across a network, stored to a file etc... I can append strings to it, 16 bit ints, 24, 32 or even make up my own.

In my particular situation I don't really need to deal so much at the bit level, but the byte level. The end server app I am communicating to basically waits for a byte, it checks the value of that byte, then calls up the proper packet handler, then the packet handler will do it's thing. In the cases where there is dynamic data, it will normally read 2 bytes, which is the size of the data that is coming. Then read that many bytes, then perhaps read another 2 or maybe even 4 bytes that could be another value and so on.

The key is being able to send a specific byte value to the server, but also be able to deal with dynamicly sized data.

I think I'll just go ahead and write my own thing and yeah unsigned int may be better choice, figured byte in C# might basically be the same.

Just thought maybe something like this already existed and I could download it or something. System.Text.ASCIIEncoding.UTF8 seems to do what I want as far as converting goes. I'll probably end up using it as the last conversions step.

 

[mikeblas]

.NET has serialization features; you should look into that. It'll do what you want, unless you need to have binary compatibility with an existing reader, or have readers hat are not .Net implementations.

 

[Whatsisname]

Also, if you are concerned about the speed of List<byte>, you are concerned about the wrong stuff.

 

[Red Squirrel]

Also, if you are concerned about the speed of List<byte>, you are concerned about the wrong stuff.

I was just asking, does it have any performance impact, or not? To use that vs byte[]. My program needs to process 100's of MB of data in miliseconds, every single small optimization counts. The portion I'm working on is a critical part.

 

[Whatsisname]

It won't have any performance impact vs your likely half baked implementation.

And if it still doesn't work, it would still probably be easier and cheaper to just call dell, buy their most expensive machine, then run it on that.

And I don't mean to insult you when I accuse your implementation of being half baked. The .NET engineers are probably better developers when it comes to these core computer-sciency problems. Use what they've done and benefit from their hard work, and use your familiarity with whatever domain you are dealing with to solve that problem. Don't try to solve the problems they already have. I always use standard libraries whenever possible, because I know that when I have to do it myself, it's going to be half baked or worse, and I'm pretty good.

 

[mikeblas]

I was just asking, does it have any performance impact, or not? To use that vs byte[]. My program needs to process 100's of MB of data in miliseconds, every single small optimization counts. The portion I'm working on is a critical part.

If you process 100 megabytes of data in 10 milliseconds, you're thinking that you're going to process 10 gigabytes of data per second. You're not going to be able to do that in C# on a single computer. It's going to be tough to do it with the inefficient C++ code you've shown in this thread (or was it the other one?), as well.

 

[Wiseguy2001]

Before I get into this, does such a class already exist?

Yes, it is called a String. This is a very efficient collection of characters, I understand where you are coming from by converting this to byte's when you are sending this over a network, but this extra conversion really isn't doing you any favours. The beauty of utf8 is that it is designed to be as compact as possible, single byte for the first non-Latin characters (same as ASCII), but (as mentioned before) the size will grow to handle more exotic characters - Is this extra overhead needed?

Believe it or not, but you are not the first to send text over the internet... If you are working with text use a text writer - That is what it is designed for.

I was just asking, does it have any performance impact, or not? To use that vs. byte[].

Using a list<byte> is about the worst way to do this:

1. If you do not define the size on creation, it will grow dynamically (overhead)
2. Regardless of what the compiler will let you do, List<> doesn't handle primitive type. Each item in the list will be boxed into an object on-the-fly (there is a large amount of overhead involved).
3. It won't handle utf8! you would need at least uInt16 (this is still only a subsection of utf8). By doing this you are losing the variable sizing of utf8.

WriteInt32 which will write an int into 4 bytes of space in it's binary value.

Cast(Uint32) will do this, but again - Just stick with text!

As for choice of language, a managed language (C#) will always have some performance overhead (the amount varies) over native (C++), but good code in C# will for faster than bad code in C++.

My program needs to process 100's of MB of data in miliseconds, every single small optimization counts. The portion I'm working on is a critical part.

May I ask what you are doing to require such high-end performance demands? And what systems are you planning on running this + what network interfaces and storage?

 

[Red Squirrel]

This is for a game server, basically before the way it worked is they serialized all changes in one shot, which required to halt game play (it's not multithreaded). As the world grows, the saves take longer and longer requiring the game to halt longer. The system is really not scalable. The way it saves is it serializes all game items to disk, which is fine due to the type of data it has (would be hard to structure for a SQL DB or something) but the issue is it saves the whole thing each time.

I redesigned the storage engine where it only saves world changes since last save, up to a limit (ex if too much changed at once). It still uses serialization as it would be too huge of a project to completly change it, but it only saves the objects that change. It saves them to a temp memory buffer (fast - needs to happen in less then 100ms or users will feel lag), then another thread takes over and saves to disk over the network to a custom C++ storage app. Then the cycle continues. When it does a memory save, depending on how many changes occurred since the last save (could be 5 seconds away, or over a minute) then it could be in the 100MB range quite easily, especially when there are a lot of players on.

My issue is when I try to formulate the network packets to send them off, I can't do it properly because of strings having characters more then 8 bit. A typical packet might look something like this:

1 byte: packet ID
4 bytes: container ID
4 bytes: serial
4 bytes: data size
[variable string]: data

Now in C++ I would use my bitstream class to serialize a string with all the data, then send it over the network. I can't seem to figure out a way of doing this in C# due to strings being UTF16 and not UTF8.

In C++ it would look something like this:

BitStream tosend;
tosend.WriteInt8(0x3);
tosend.WriteInt32(contid);
tosend.WriteInt32(serial);
tosend.WriteInt32(data.size());
tosend.WriteString(data);

connection.Send(tosend);

I was going to write a class using a List of bytes so the save process could serialize to that, then I could also send the packet this way, but as suspected it may not be very efficient. One thing I noticed in C# is loops take a very long time compared to C++ and I would require to use loops a lot if I wrote such a class.

And what do you mean by UTF8 growing? I thought UTF8 meant, 8 bit characters? Also I'll check out text writer that might be what I need if I can somehow convert it to a byte array (which the network class takes)

 

[Lunknumb]

UTF-8 is only 8 bits per char up to 0x7F (basicly, inside ASCII), above that there's 16 bits per char.

 

[Red Squirrel]

Hmm did not know that, then what does a C++ string use? I'm pretty sure it's always 8 bit per char aka standard ascii table.

I've been looking at TextWriter and I think it may work how I want, only issue is there does not seem to be a way to read from it other then ToString() which does some weird stuff that is based on regional settings so that wont really work for what I want.

 

[Lunknumb]

STL uses these:

std::string is UTF-8 with platform dependant codepage (chars are 8-16 bits).
std::wstring is UTF-16 (chars are 16-32 bits), which is equivalent to the .NET System.string class

 

[Red Squirrel]

Interesting guess I've been lucky so far in my usage of string then as I've never run into any issues where I get some weird output when sending over network or what not.

 

[aL Mac]

I didn't really read your whole post... but are you aware in c# there is an Encoding class and several formats, I believe including the one you wanted.

 

[Red Squirrel]

I've been thinking, for my packets think I will just use regular byte arrays for the packet header themselves, given in most cases I already know how much chars I need so using that is not a problem, then I will just convert the string (which is the data being sent) to byte array at that point. I knew about the encoding that would do a conversion, but I was hoping to just drop string out of the equation completely, and write my data directly to a stream that is strictly 8 bit chars, then not have to worry about any kind of conversion, but either way, throughout the app there's lot of strings and there is no way I could change that part so no matter what I have to convert stuff, just a question of where. So yeah guess I'm better off sticking to strings, and then just do one single conversion at the very end of the process.