Email sanitizer-extractor in Lua

Yesterday a friend asked me to write a little script that reads a file and outputs every email it reads in another file, discarding any duplicates. After making the program he told me that he was searching the Internet and couldn't find something similar so I should upload it somewhere just in case someone else needs it.

To make it more interesting I managed to make it a single call program: everything is defined inside the arguments of a single call. In fact there are two calls: the first returns an object, a function of which is immediately called. Anyway here is the code:

io.open("output.txt","w"):write((string.gsub(" "..io.open(((arg or {})[1] or "input.txt")):read("*a").." ",".-([%w%.%-_]+@[%w%.%-_]+).-",function (email) email=string.lower(email) print("EMAIL: '"..email.."'") emails=emails or {} for index,emailseen in ipairs(emails) do if emailseen==email then return "" end end table.insert(emails,email) return email.."\n" end)))

Now let's break that up and put some comments, shall we?

io.open("output.txt","w"):write( --io.open opens a file in write mode and returns the file handle which instead of being stored in a variable is immediately used by calling it's filehandle:write function.

(string.gsub( --string.gsub will finally return two arguments: the email list, one per line, lowercase, and without duplicates and the number of replacements it did. The first argument is our output. I can discard the second by putting the function call, therefore the returned argument list, into parentheses. In Lua print((5,"kostas","klapatsimpala")) will just print 5.

" "..io.open(((arg or {})[1] or "input.txt")):read("*a").." " --This is the first argument to string.gsub. like we did before, we open a file in read mode and immediately use the returned handle to do a full read of the file. A tricky part is the "(arg or {})[1] or "input.txt"" part. If you call a lua script with extra arguments then the arg table will be created by Lua. If it exists then the "arg or {}" part will evaluate in "arg" (if on the left side of an "or" is a true value then "or" simply results in that) and then "(arg)[1]" will return the first variable which is a custom input filename. That filename "ORed" with "input.txt" will simply return that filename (since any strings are true values for Lua, so OR will evaluate in the left argument). If you didn't call the script with any arguments then the arg table will not exist, thus the "(arg or {})" part will result in a newly created empty table. Of course if you index it's first cell you'll find nothing, so the "({})[1] or "input.txt"" will result in "input.txt" (if "or" finds a false or nil value on it's left it will simply return the value on it's right). Finally I add two space characters: one to the start of the read data and one to the end. These are added so that the pattern matching I use will apply to any emails exactly at the beginning or exactly at the end of the read data.


,".-([%w%.%-_]+@[%w%.%-_]+).-" --The second argument is the pattern. I am breaking up the whole text in the following way: any number of any characters (as less as possible) followed by any number of email allowed characters (as much as possible), followed by @, followed by any number of email allowed characters (as much as possible), followed by any number of any characters (as less as possible). The "email allowed characters" are: alphanumerics, dot, dash, underscore). From this pattern I want to capture just the email part.

,function (email) --Now this is the best part. An anonymous function. It is created without being stored in a variable (which would give it a name) and immediately used as an argument to string.gsub. This function accepts a single argument: email. string.gsub will call it for every match with the capture (the email) as an argument.

email=string.lower(email) --First of all we turn the email to lowercase

print("EMAIL: '"..email.."'") --Debugging message...

emails=emails or {} --Remember what we said. If the left argument is true (not false and not nil) then it is returned, so if the emails variable has already been defined nothing will happen because emails=emails will be executed. If the emails variable is not defined (is nil) though, then "or" will return it's right argument therefore emails={} will be executed and emails will be initialized as an empty table.

for index,emailseen in ipairs(emails) do --For every already seen email do:

if emailseen==email then return "" end --If this already seen email is the same with the new capture then just return "" so that the whole match will be replaced by nothing. Remember that although the capture is just the email, the match includes the email as well as the preceding and the following characters.

end --end for.

table.insert(emails,email) --If we managed to get here then this email capture is seen for the first time. We insert it in the emails table.

return email.."\n" --and finally we return the email capture (lowercase) followed by a newline. This will replace the whole match.

end --end of the anonymous function

) --closing of string.gsub

) --that's the second parentheses for string.gsub (to discard the second returned argument)

) --closing of write.


That's all. I seem like it is working but I haven't done any extensive debugging.

AskWise v1.0.0

I finally finished rewriting AskWise and it think it's time to take it out of beta. It is a menu driver console application that using databases with sets of known numerical data, predicts missing values in other sets of numerical data. For example, one of the included databases has some sets of height, weight, age and sex (male=0 female=10). Given this database, AskWise can predict someones sex given his height, weight and age. In fact you can give it whatever values (of these four) you know for a person and it will try to predict the rest of them. This database is not useful but you can create your own databases and use them with AskWise.

That's the fourth time I program AskWise's algorithm. The first one was during my final year in school when me and a friend of mine were challenging each other to find functions with specific strange graphs. I though I should try to make a function that can easily be adapted to connect any number of points, so I made one. Later I realized that I could use this function as a quick and dirty way to predict missing values in a set so I wrote a program in PLua that used it for this purpose. I later rewrote it in Visual Basic improving the algorithm a little. Then I rewrote it in Lua, further improving the algorithm and now I wrote it again in Lua, using almost the same algorithm but with a better, menu driven UI.

The new version is released under the terms of the GNU/GPLv3, works on both Windows and Linux and can be found here.

I also wrote some documentation for it, where I try to explain both how to use AskWise and how it makes it's predictions. It was quite hard to do this in English and I believe that my syntax might be quite complex in some parts. If you believe that something is lacking clarity (I'm sure there are such parts) then, please, point it out to me.

PS: This is out of beta but there might still be bugs. If you find one, AskWise will create a small bug report on it's directory, I would be glad if you send it back to me. :-)

Long absence (or: my news)

I am sorry for not posting during the last months but I was quite busy. I had to do some projects for my university; this is a quite exciting year. :-P Perhaps the most interesting project of them is the one for Microcomputers: Along with my partner we decided to make a PIC based Morse decoder, decoding Morse from it's microphone and displaying characters on an LCD. Well we still have problems making the PIC programmer which for some reason doesn't work.

I also work on another program that produces an exam period schedule based on what subjects each student will take an exam on, so that students have enough days between the subjects to study. It works very well but, although it scales linearly, I find it quite slow for big input data, roughly estimating that it will need 2 days to optimize the exam schedule for a school of 3000 students. So I am still working on improving it's speed. I will post this program here when I finish it.

I am also rewriting AskWise from the scratch. It will be a menu driven console program and I hope that I won't consider it a beta anymore when it's finished.

Finally I was going to rewrite for the new Intrepid Ibex my old Vaio & Hardy Heron how-to but I don't have to, since Shawe wrote an excellent how to. It is in Spanish but you can easily understand it even without translating it.

So I guess my next post will be in less than a month. :-)
Have fun.

Tiny Lua Distribution 5.1.4

Lua 5.1.4 is out so I recompiled it with Dev-C++ to make the windows version a standalone EXE. Strangely enough, it became some KB larger than 5.1.3. Anyway here is the new Tiny Lua Distribution. (contains Linux and Windows interpreters)

Big table implementation in Brainfuck

In the past I had found an efficient way of implementing tables on the Brainfuck array. There were already some table implementations which used at least 2*n Brainfuck cells for storing an n sized table. My implementation needed only n+4.

Also, all known implementations had a limit of 256 cells (Or equal to the size of a cell on the Brainfuck array. For the rest of the post I will assume that the cell size is one byte).

During my road-trip to Spain, based on my old implementation, I found a new one that can use multiple indexes to implement tables of ANY size on the Brainfuck array. Additionally the table cell size can be any multiple of the Brainfuck array's cell size.

It uses a format similar to my old implementation: a head moving through the table while decreasing the index it carries, passing table data over it, until it finds the correct position,the it does it's job (read/write) and then it returns in the same way but using a copy of the index.

For a table with n cells of size k each the form of the header is:

• k empty cells: When the head need's to move to the right, it will have to move the first k cells on it's right in these empty cells before moving itself k positions to the right (thus moving itself to the next indexed position)
• n index cells: Before starting a read or write operation you will have to store the index here. The leftmost index is the most important. For example if the maximum possible value for an index is 9 (instead of 255) and you use two indexes, to access the 15th cell you would have to use 1 for the left index and 4 for the right index (not 5 because the numbering starts from 0).
• k data cells: To write to the table you will have to put the data in these cells. If you read from the table, after the end of the read operation the read data will be here. These cells MUST be cleared before starting a read operation.
• n empty (index) cells: While the index cells are reduced, these cells are increased so that, when the seek operation finishes, they will hold a copy of the original indexes which will be used for returning the head to the beginning of the table.

After the header the table data is assumed to be laying. n*k cells in total. For example for a size 400 table with 2 bytes per cell they would have this form:
Data 1 of index 0,0
Data 2 of index 0,0
...
Data 1 of index 0,255
Data 2 of index 0,255
Data 1 of index 1,0
Data 2 of index 2,0
Data 1 of index 1,1
Data 2 of index 2,1
...
...
Data 1 of index 1,143
Data 2 of index 1,143

So you will need 2*k+2*ceil(log256(n))+n*k cells on the Brainfuck array to store a table with n, k-sized cells. (ceil(log256(n)) is the number of the indexes)

Therefore the code needed to write or read from the table depends on n and k. I actually wrote some metacode that provided with n and k produces the corresponding code. I then turned the metacode into a Lua script. It's not well written but I believe that it works. I tried the following cases:

1. Writing two values to a table with n<256 and k=1 and then reading them back.
2. Writing two values to a table with 256<n<256^2 and k=2 and then reading them back.

and everything worked fine.

If you want to try the script yourself you download it from here.

Here is some example code:

n<256 k=1
Write:
>[->>+>[-<<<<+>>>>]<[->+<]<[->+<]
<[->+<]>]>>>[-]<<[->>+<<]>[-[-<+>
]<<<<[->>>>+<<<<]>>>]<<<
Read:
>[->>+>[-<<<<+>>>>]<[->+<]<<[->+<
]>]>>>[-<<+<<+>>>>]<<<<[->>>>+<<<
<]>>>[-<[-<+>]>[-<+>]<<<<[->>>>+<
<<<]>>>]<<<

256<n<256^2 k=1
Write:
>>[->>>+>[-<<<<<<+>>>>>>]<[->+<]<
[->+<]<[->+<]<[->+<]<[->+<]>>]<[-
>>>+>>[-<<<<<<+>>>>>>]<[->+<]<[->
+<]<[->+<]<[->+<]<[->+<]>>-[->>>>
[-<<<<<<+>>>>>>]<[->+<]<[->+<]<[-
>+<]<[->+<]<[->+<]>>]<]>>>>>[-]<<
<[->>>+<<<]>>[-<[-<+>]>[-<+>]<<<<
<<[->>>>>>+<<<<<<]>>>>>]<[-[-<+>]
>[-<+>]<<<<<<[->>>>>>+<<<<<<]>>>>
>-[-<[-<+>]>[-<+>]<<<<<<[->>>>>>+
<<<<<<]>>>>>]<]<<<<
Read:
>>[->>>+>[-<<<<<<+>>>>>>]<[->+<]<
[->+<]<<[->+<]<[->+<]>>]<[->>>+>>
[-<<<<<<+>>>>>>]<[->+<]<[->+<]<<[
->+<]<[->+<]>>-[->>>>[-<<<<<<+>>>
>>>]<[->+<]<[->+<]<<[->+<]<[->+<]
>>]<]>>>>>[-<<<+<<<+>>>>>>]<<<<<<
[->>>>>>+<<<<<<]>>>>>[-<<[-<+>]>[
-<+>]>[-<+>]<<<<<<[->>>>>>+<<<<<<
]>>>>>]<[-<[-<+>]>[-<+>]>[-<+>]<<
<<<<[->>>>>>+<<<<<<]>>>>>-[-<<[-<
+>]>[-<+>]>[-<+>]<<<<<<[->>>>>>+<
<<<<<]>>>>>]<]<<<<