![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
Best language for text processing and manipulation?
- Thread starter OldM3ta
- Start date
It depends on your programming experience and the exact nature of the parsing you want to do. Perl is good for simple tasks, but it is slow and not adept at more complicated parsing. Traditional languages like C++ are harder to learn, but much faster. C# is slower than C++ but faster and more powerful than Perl; C# takes some of the edge of the learning curve.
Lots of dependencies affect your decision, so it's hard to recommend a language without knowing more specifics.
Lots of dependencies affect your decision, so it's hard to recommend a language without knowing more specifics.
I've been dealing with this lately.
I started a new job and they told me to come up with a report based on some log data. Long story short, I redid the work several times.
For me it was important that the solution be cross platform, is that an issue for you?
PERL and Python have been serving me well in that regard. Initially bash worked pretty well too (but not truly cross platform - though it mostly works in cygwin)
I started a new job and they told me to come up with a report based on some log data. Long story short, I redid the work several times.
For me it was important that the solution be cross platform, is that an issue for you?
PERL and Python have been serving me well in that regard. Initially bash worked pretty well too (but not truly cross platform - though it mostly works in cygwin)
thefreeaccount
Gawd
- Joined
- Aug 8, 2010
- Messages
- 832
Whaa? I'd expect a C++ program written using strings & iostreams to be at least 10x slower than an equivalent program in perl.
There's an interesting old thread here that you might want to read through.
(And I apologize for djn.homeunix.net being down at the moment so my old code samples aren't accessible. I'll get it back up ... some day. The machine is in another end of the country, and I didn't grab those files from it before moving.)
edit: Long shot. Mikeblas, do you think you have my program from that lying around? (sqlfix.zip or .cpp , I guess).
Last edited:
Whatsisname
[H]F Junkie
- Joined
- Nov 15, 2000
- Messages
- 10,201
I would normally pick Perl because it is often faster in terms of Programmer time.
I would expect a well written C/C++ program to be faster cpu wise, but for the majority of parsing tasks the extra programmer time isn't worth it IMO.
And I wouldn't expect the C/C++ programs to be slower than perl. If you are going to do an intense parsing task, anyone with any kind of sense won't use C strings.
I would expect a well written C/C++ program to be faster cpu wise, but for the majority of parsing tasks the extra programmer time isn't worth it IMO.
And I wouldn't expect the C/C++ programs to be slower than perl. If you are going to do an intense parsing task, anyone with any kind of sense won't use C strings.
Well - C strings are not that bad. C++ std::string probably have more overhead; no idea how much.
The result in the thread I linked was that the fastest result was from a C++ program (written as C with classes, not idiomatic C++) treating the file as a bunch of bytes.
another vote for Perl from me
I'm not even that experienced with it, was only given a brief run through in two lectures in a Unix Admin course, everything else was just googling stuff as I went
but even then I can still whip up stuff like parsing programs with much less hassle than with other languages im more experienced in
I'm not even that experienced with it, was only given a brief run through in two lectures in a Unix Admin course, everything else was just googling stuff as I went
but even then I can still whip up stuff like parsing programs with much less hassle than with other languages im more experienced in
Whatsisname
[H]F Junkie
- Joined
- Nov 15, 2000
- Messages
- 10,201
For that task I think C strings would have been fine, because you're not doing lots of string chopping/concatenating/reordering and shuffling them around, which is where c strings really start to get annoying.
If you were doing some kind of operation where you were transforming an xml file or something in some big way, then C strings would be pretty slow compared to a pascal type string where the length is stored along with the string, or some other improved string class.
If you were doing some kind of operation where you were transforming an xml file or something in some big way, then C strings would be pretty slow compared to a pascal type string where the length is stored along with the string, or some other improved string class.
You do have the option of passing the length around together with the string to work around that. Annoying, of course - but the C string format itself isn't a big slowdown.
(That said, I'm personally fond of mmapping the entire input file and using offset+length within it.)
Perl/Python are definite contenders if you want flexible reporting tools but don't neglect the traditional unix awk/sed/grep tools. If your processing is CPU-bound, keep in mind that a unix pipeline gives you automatic parallelization. Also, the mawk implementation of awk (which is standard with Ubuntu these days) is blazingly fast - in some cases as fast or faster than the equivalent C++ implementation.
Why so? Are you referring to any program in general, or a program specific to the OP's needs (which, at this point, you don't know)? I'm not sure how you can formulate such an assertion, not to mention find a reason to stick by it. What tests have you run to substantiate your opinion?
I can try digging around, but it's pretty unlikely. Did you send it to me in email? Do you remember which address I was using for that thread? Or was I making people give download points?
Download points, I'm afraid.
I know where I have a copy - I'll just grab it next time I visit my parents. Don't stress, I was just wondering if you happened to have every single forum-related thing lying around sorted by year, or something.
Download points, I'm afraid.
I know where I have a copy - I'll just grab it next time I visit my parents. Don't stress, I was just wondering if you happened to have every single forum-related thing lying around sorted by year, or something.
Not quite that crazy, sorry. I might have a backup of the machine I was using around, if I could remember what machine name was my main home dev rig at that time ...
thefreeaccount
Gawd
- Joined
- Aug 8, 2010
- Messages
- 832
The OP said he needed a parser that doesn't do anything - it just stops parsing when it hits the right expression. The overhead of calling a bunch of virtualized IO functions and encapsulating incoming text into objects is going to be huge for a parser that isn't actually doing anything with its tokens, even if it happens to have quite a few states.
I can't be the only one who feels that this is obvious because, while several posters in the thread you linked used OO languages, only one person bothered to try breaking the problem down using a proper OO approach. OTOH, it looks like he was the only one to submit a correct program. 8)
There is an argument to be made that perl is an OO language with backwards compatibility like C++, but IMO that would be incorrect because perl does not have an OO library.
Hi guys, finally had some time to respond to all this great input.
I am traditional C++ programmer. I have worked in the past briefly with Perl and Python. The need for the program is to be run on a large text file, seeking a keyword or keyphrase to use to truncate the file at that discovery point. Speed would be good. It would run from a a Windows environment with access to Cygwin.
This is what I've heard, as well. That Perl is a dog to read and write. That Python had it beat in almost every way.
Thanks, I'll have to keep that in mind, except I'm not familiar enough with them. I would guess I would create a script to run that has the write command calls for the shell I'm in? And then could even possibly use them in Cygwin?
So what is your recommendation thefreeaccount?
I am traditional C++ programmer. I have worked in the past briefly with Perl and Python. The need for the program is to be run on a large text file, seeking a keyword or keyphrase to use to truncate the file at that discovery point. Speed would be good. It would run from a a Windows environment with access to Cygwin.
This is what I've heard, as well. That Perl is a dog to read and write. That Python had it beat in almost every way.
Thanks, I'll have to keep that in mind, except I'm not familiar enough with them. I would guess I would create a script to run that has the write command calls for the shell I'm in? And then could even possibly use them in Cygwin?
So what is your recommendation thefreeaccount?
He says it'll tokenize, then stop. Tokenization is work. Looks like might not be what he meant, but it certainly is what he said before your post. He might have meant scan and stop, and that's entirely different. Either way, I see no advantage to Python or Perl aside from some ease of development, but even that is conditional because of the limitations inherent in those languages.
It won't be huge, but it can be important. But who said anything about virtual I/O? I think all the I/O the program would do is physical. Or do you mean virtual functions? The indirection of a virtual function isn't huge by any stretch of the imagination.
People are often wrong in groups.
You're simply, then, looking for a string within a file? What of the tokenization you mentioned earlier?
Good catch! Unfortunately, no backups of that machine remain.
Tokenization may or may not be necessary. Can you give an example of what it is you want?
Going the other way, if you want this file to end when "bar" is encountered:
Code:
Fooey
Bunky
/* bar */
Zing
bar
Blap!then you don't need to do tokenization. If you want the file to end when the first "bar" that is not a comment is encountered, then you need tokenization to figure out if you're in a comment token, or not. Speaking semantically, of course -- your eventual implementation may or may not actually tokenize, so long as it implements that desired behaviour.
Tokenization implies that you're eating the content of the file and breaking the content into pieces -- tokens -- so that you know the state of some grammar in the file. Here, it's simply the grammar of eating "/* ... */" comments. If you don't need that, then the code you write is necessarily simpler and probably faster.
GlimmerMan
Gawd
- Joined
- Feb 13, 2005
- Messages
- 982
For purely text processing work, I've used sed/awk, Perl, and Python. While Perl programs are fun to write, they are hard to read after some time as passed. Python is now my go-to language for text processing, mostly because I find Python programs are much easier to maintain.
On occassion, I have used Boost.Regex in C++ programs that has to do some text processing but is not the primary task of the program.
On occassion, I have used Boost.Regex in C++ programs that has to do some text processing but is not the primary task of the program.
If you're too dumb to think it through, sure.
At the risk of threaddrift an unwarranted flaming, I've done text manip in python and perl..while python was a bit more organized (has to be, by nature) - I've found that perl was easier to code in. Maybe if you're coming from a background with no scripting/development it would be a better choice, I've just had better success with perl.
Coming into python having done some bash, perl, ruby etc the forced organization actually made me "stumble" (lack of a better word at the moment) more than anything because I like to indent etc in my own fashion.
As to the problem if you're just reading the file I guess the overall size of the file needs to come into play as well - did I miss that at some point in the thread?
Without knowing all the ways this could be used/size of the file...If I were coming at this problem I'd probably read the file line by line - if it didn't contain the matching string - write that to another (assuming they want the original file preserved) file, if it did contain the matching string - check that the line didn't start with the comment char - if it did, continue on if not close the file.
untested off the top of my head - assumes '#' is the comment char
I'm no perl expert though, that's just how I'd start my go at it. I'd have to put more thought into it if multiline comments (ala php) came into play.
Coming into python having done some bash, perl, ruby etc the forced organization actually made me "stumble" (lack of a better word at the moment) more than anything because I like to indent etc in my own fashion.
As to the problem if you're just reading the file I guess the overall size of the file needs to come into play as well - did I miss that at some point in the thread?
Without knowing all the ways this could be used/size of the file...If I were coming at this problem I'd probably read the file line by line - if it didn't contain the matching string - write that to another (assuming they want the original file preserved) file, if it did contain the matching string - check that the line didn't start with the comment char - if it did, continue on if not close the file.
untested off the top of my head - assumes '#' is the comment char
Code:
#!/usr/bin/perl
open IF, "<inputfile.txt"; #input file, the file you're reading
open OF, ">>outputfile.txt"; #output file, the file you're writing to
$word="whatever"; #word we're looking for
while (<IF>){
if ($_ =~ /^[^#]+$word/){
print OF $_ . "\n"; close OF; close IF; die "Pattern matched";}
else{print OF $_ . "\n";}
}
close IF; close OF;I'm no perl expert though, that's just how I'd start my go at it. I'd have to put more thought into it if multiline comments (ala php) came into play.
Last edited:
Sounds like a job for awk...
Perl is just Perl. It's a bunch of magic that lets you write code however you damned well feel like. The OO bits of Perl are clearly bolted onto the side rather than being a fundamental part of the language design.
Nothing wrong with Perl or Python; the problem is one-word answers.
The only functional problem is that you're processing line by line. The only performance problem (other than using an interpreted language) is that you're using regular expressions where string matching will do.
So (I'm still new at this and honestly asking, not being a smartass) what would be the better way to go about this?
I tend to use regex a lot in and out of perl simply because I know it...but if there's a better way, I'm game.
A regex needs to parse the regex, worry about all the wildcards and repeaters, and so on. If you need all those features, it's great. Here, I don't think we do -- we still haven't heard back from the OP with clear specifications.
If we don't need those features, seaching for a simple substring is lots faster. You can do that with the index function.
Again, we don't know if the target text might span lines, or not. If it does, then you'll need to find some way to read from your file a block at a time -- or to read a line at a time, but keep enough state around so that you can match across read lines.
If we don't need those features, seaching for a simple substring is lots faster. You can do that with the index function.
Again, we don't know if the target text might span lines, or not. If it does, then you'll need to find some way to read from your file a block at a time -- or to read a line at a time, but keep enough state around so that you can match across read lines.
I'm actually doing some benchmarks here to see just how much of a performance hit there is. In python, a precompiled regex match takes 8-10 longer than string equality, where a substring match is only about twice. Granted, these are with very simple & short strings but it's enough to remind me just how much I don't really want to use regular expressions unless I actually need to match complex patterns.
Eh, there is a niche for regexps. They're decent for pulling out parts of strings, or checking for matches against simple patterns; things where writing parsing code would be more work but not easier to read. Admittedly, that's a small subset of what people seem to be doing with them.