texts:
I'm currently attempting to learn OCaml, and I'm working thought the Project Euler problems to do so.  Here's some code I knocked together for problem 10.

I am looking for idiomatic feedback rather than algorithmic

(*
The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17.

Find the sum of all the primes below two million.

MODIFYING CODE FROM PROBLEM 7

*)

(* first thing we are going to do is write a bit of code that checks to see if a number is prime   *)

let rec isPrimeRec number start = if (start*start)>number then 1 else  if number mod start = 0 then -1 else isPrimeRec number (start+1);;

let i = ref 2;;
let sum = ref 0 in
while !i <2000000 do
if (isPrimeRec !i 2) = 1 then
begin
sum:= !sum + !i; i:= !i + 1;
Printf.printf "%d is prime, it is prime number %d\n" !i !sum;
end
else
i:= !i + 1
done;;

let temp = ref 0;;
temp:= isPrimeRec 19 3;
Printf.printf "The value is %d\n" !temp

Now - I almost purposely didn't write this to be efficient algorithmically (for example I am aware that prime numbers can't be even)  and there are a few other things that I would change for efficiency - but I'm interested in style feedback - so I'd like the code critiqued much more on the level of "In OCaml, one would normally bracket expression X for readability" or "OCaml let's you use this, clearly syntax instead" - rather than "it's a property of prime numbers that"


texts: Program for binary to decimal and vice versa

texts:
I started to learn C a week ago and this is my culminating program of what I've learned but I could use a few pointers.

#include <math.h>
#include <stdio.h>
void binary(int n){
    int x = 0, m = 0, z = n;
    while(n >= 1){
            ++x;
            n /= 10;
            }
    n = z;
    int i[x];
    x = 0;
    while (n != 0){
            ++x;
            i[x] = n % 10;
            n /= 10;
            }
    while(x >= 0){
            if (i[x] == 1){
                    if (x == 1)
                            ++m;
                    else m += pow(2,x - 1);
            }
            --x;
    }
    printf("%d", m);
}
void convert(int n){
    int i[16];
    int j, k, p;
    for(k = 15; k >= 0; --k){
            i[k] = n % 2;
            n /= 2;
            }
    int *a, *b;
    a = &i[0];
    b = &i[16];
    for( ;*a == 0; ++a)
            ;
    while(a != b){
            printf("%d", *a);
            ++a;
    }
}
int main()
{
    int n;
    char m;
    printf("\nType b for binary to decimal conversion or d for decimal to binary conversion (max 65535)\n");
    m = getchar();
    printf("\nPrint number to be converted\n");
    scanf("%d", &n);
    if (m == 'b'){
            printf("%d converted to a decimal number is ", n);
            binary(n);
    }
            else{
                    printf("%d converted to a binary number is ", n);
                    convert(n);
            }
}


texts:
I have some code:

  $matchingEvents = [];
  foreach( $this->recurringEvents as $event ){
    if( $event['RCE_Type'] === 'Month' && $event['RCE_Day'] === $dayOfMonth ){
      array_push( $matchingEvents, $event );
    } else if( $event['RCE_Type'] === 'Week' && $event['RCE_Day'] === $dayOfWeek ){
      array_push( $matchingEvents, $event );
    }
  }

Which duplicates the line:

array_push( $matchingEvents, $event );

This is small duplication, but still duplication. I could reformat it as this:

  $matchingEvents = [];
  foreach( $this->recurringEvents as $event ){
    if( ( $event['RCE_Type'] === 'Month' && $event['RCE_Day'] === $dayOfMonth )
        ||
        ( $event['RCE_Type'] === 'Week' && $event['RCE_Day'] === $dayOfWeek ) ){
      array_push( $matchingEvents, $event );
    }
  }

But I feel like this drastically reduces the readability of the code.

In this type of situation, is DRY or simple more important?


texts:
I've made a quick finally type in C++:

template<class F>
class finally_type {
public:

    explicit finally_type(F f) : function(f) {}
    ~finally_type() { try { function(); } catch (...) {} }

private:

    F function;

};

template<class F>
finally_type<F> finally(F f) { return finally_type<F>(f); }

(Edited to add swallower to prevent exceptions from propagating out of the callback.)

With C++0x, it's convenient to use this class as follows:

void test() {

    int* i = new int;

    auto cleanup = finally([i]() { delete i; });

    may_throw();

}

Regardless of whether may_throw() throws, the finally ensures that the cleanup code executes when the stack is unwound. The equivalent idiom in C++03 is using a local type:

void test() {

    int* i = new int;

    struct finally {
        finally(int* i) : i(i) {}
        ~finally() { delete i; }
        int* i;
    } cleanup(i);

    may_throw();

}

As you can see, the closure must be managed explicitly, with loads of repetitive (and consequently error-prone) code; with a lambda, this process is vastly simplified.

Frankly, I don't see much use to it, since if you're interfacing with legacy code, you're probably just going to wrap the types or use a shared_ptr with a custom deleter. You could use it to manage low-level locks in multithreaded code, if that's your fancy for some reason.

Anyway, assuming someone does find it useful, is there anything important that I missed?


texts:
This is a Leetcode problem -

  Given a string containing just the characters '(', ')', '{', '}', '[' and ']', determine if the input string is valid.

  An input string is valid if -

  Open brackets must be closed by the same type of brackets.
  Open brackets must be closed in the correct order.

  Note that an empty string is also considered valid.

  Example 1 -

Input: "()"
Output: True

  Example 2 -

Input: "()[]{}"
Output: True

  Example 3 -

Input: "(]"
Output: False

  Example 4 -

Input: "([)]"
Output: False

  Example 5 -

Input: "{[]}"
Output: True

Here is my solution to this challenge -

def is_valid(s):
    if len(s) == 0:
        return True
    parentheses = ['()', '[]', '{}']
    flag = False
    while len(s) > 0:
        i = 0
        while i < 3:
            if parentheses[i] in s:
                s = s.replace(parentheses[i], '')
                i = 0
                flag = True
            else:
                i += 1
        if len(s) == 0:
            return True
        else:
            flag = False
            break
    return False

So I would like to know whether I could improve performance and make my code shorter.


texts:
I have spent hours on this code. I am a beginner and open to advice and improvements. I've put # where I feel insecure about the code.

It lets you pick your marker, checks winner, checks draw, ask player if want to play again and checks if the square is occupied.

#TicTacToe
def drawBoard(board):
      print('   |   |')
      print(' ' + board[7] + ' | ' + board[8] + ' | ' + board[9])
      print('   |   |')
      print('-----------')
      print('   |   |')
      print(' ' + board[4] + ' | ' + board[5] + ' | ' + board[6])
      print('   |   |')
      print('-----------')
      print('   |   |')
      print(' ' + board[1] + ' | ' + board[2] + ' | ' + board[3])
      print('   |   |')

game_board = ['#', ' ', ' ',' ', ' ',' ', ' ',' ', ' ',' ']

player1=''
player2=''

def setplayersmove(playermarker):
    print("Select your marker: ")
    playermarker = input(">  ")
    return playermarker

def makemove(someone):
    print("Enter a number 1-9 for your move: ")
    movenum = int(input('> '))
    if movenum < 1 or movenum > 9:
        print("Number between 1 and 9 only.")
        makemove(someone)
    else:#makes sure the square is not occupied by ANYONE
        if game_board[movenum] != player1 and game_board[movenum] != player2:
            game_board[movenum] = someone
        else:
            print("Move already occupied. Try again: ")
            makemove(someone)

def tryagain():
    print("Try again?: Y/N")
    ans = input('> ').lower()
    if ans == 'y':
        gamestart()
    else:
        exit(0)

def checkwinner():
    if game_board[1] == game_board[2] == game_board[3] and game_board[1] != ' ':
        print("We have a winner!")
        tryagain()
    elif game_board[4] == game_board[5] == game_board[6] and game_board[4] != ' ':
        print("We have a winner!")
        tryagain()
    elif game_board[7] == game_board[8] == game_board[9] and game_board[7] != ' ':
        print("We have a winner!")
        tryagain()
    elif game_board[1] == game_board[4] == game_board[7] and game_board[1] != ' ':
        print("We have a winner!")
        tryagain()
    elif game_board[1] == game_board[5] == game_board[9] and game_board[1] != ' ':
        print("We have a winner!")
        tryagain()
    elif game_board[2] == game_board[5] == game_board[8] and game_board[2] != ' ':
        print("We have a winner!")
        tryagain()
    elif game_board[1] == game_board[2] == game_board[3] and game_board[1] != ' ':
        print("We have a winner!")
        tryagain()
    elif game_board[3] == game_board[5] == game_board[7] and game_board[3] != ' ':
        print("We have a winner!")
        tryagain()

def checkdraw():#checking draw. Advice welcome here...
    count = 0
    for item in game_board:
        if item != ' ':
            count += 1
    if count == 10:
        print("---Game Drawn---")
        drawBoard(game_board)
        tryagain()

def gamestart():

    global player1#ways to avoid using global?
    global player2
    global game_board

    game_board = ['#', ' ', ' ',' ', ' ',' ', ' ',' ', ' ',' ']
    drawBoard(game_board)
    print("---Player I--- ")
    player1 = setplayersmove(player1)
    print("---Player II--- ")
    player2 = setplayersmove(player2)

    gameon1 = True

    while gameon1:
        print("---Player I's Turn---")
        makemove(player1)
        drawBoard(game_board)
        gameon2 = True
        checkwinner()
        checkdraw()

        gameon1 = False

        while gameon2:
            print("---Player II's Turn---")
            makemove(player2)
            drawBoard(game_board)
            gameon1 = True
            checkwinner()
            checkdraw()
            gameon2 = False

gamestart()


texts:
I'm writing a SQLite wrapper for C++. I have this class representing a database:

  class Database {
    sqlite3 *db;
  public:
    Database(const std::string& file_path, bool readonly = false) throw(SQLiteException);
    ~Database();

    std::vector<std::map<std::string, Value> > Query(const std::string& query) throw(SQLiteException);
    std::vector<std::map<std::string, Value> > Query(const std::string& query, const std::vector<Value>& bindings) throw(SQLiteException);
  };

Where Value is a container for an int, double, std::string or std::vector<char> (because the returned type isn't yet known at compile-time).

I'm concerned about the return value types of Database::Query. How could I simplify this?


texts: Dynamically importing Python source modules from a given directory

texts:
I'm working on an open-source test framework which needs to dynamically import Python source modules from a given directory.

I can't just use importlib.import_module() or __import__(), because it's a black box and I need to do some source-code-rewriting. On the other hand, I feel the PEP 302 Import Hooks to be overkill for what I'm trying to do (they seem to be designed with importing totally foreign file formats in mind), so I rolled my own:

import os
import sys
import types

def import_module(dir_path, module_name):
    if module_name in sys.modules:
        return sys.modules[module_name]

    filename = resolve_filename(dir_path, module_name)

    with open(filename, 'r', encoding='utf-8') as f:
        source = f.read()
    # Here I do some AST munging
    code = compile(source, filename, 'exec')

    module = create_module_object(module_name, filename)
    exec(code, module.__dict__)
    sys.modules[module_name] = module
    return module

def resolve_filename(dir_path, module_name):
    filename = os.path.join(dir_path, *module_name.split('.'))

    # I happen to know that the calling code will already have
    # determined whether it's a package or not
    if os.path.isdir(filename):
        filename = os.path.join(filename, '__init__.py')
    else:
        filename += '.py'
    return filename

def create_module_object(module_name, filename):
    module = types.ModuleType(module_name)
    module.__file__ = filename

    if '__init__.py' in filename:
        module.__package__ = module_name
        module.__path__ = [os.path.dirname(filename)]
    else:
        if '.' in module_name:
            module.__package__, _ = module_name.rsplit('.', 1)
        else:
            module.__package__ = ''

    return module

To paraphrase Greenspun's Tenth Rule:

  Any sufficiently complicated Python program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of the import system.

So, does my code do everything it's supposed to? I wrote my tests against an implementation that used importlib.import_module() and then rewrote it as this one, which gives me some confidence, but I'm not sure whether I've missed any important steps.


texts: A simple shellcode that will kill all your processes GNU/Linux x86_64

texts:
So I made a simple shellcode that will kill all your process.
Now I would like to know, can it be done in some better way?
It contains 13 bytes. Thanks for all your feedback

Assembly:

 global _start

 section .text

 _start:

     xor rax, rax
     add al, 0x3e
     dec rdi
     push 0x9
     pop rsi
     syscall

Equivalent in C:

#include<stdio.h>
#include<string.h>

unsigned char code[] =
"\x48\x31\xc0\x04\x3e\x48\xff\xcf\x6a\x09\x5e\x0f\x05";

int main()
{
  printf("Shellcode Length:  %d\n", (int) strlen(code));

  int (*ret)() = (int(*)())code;

  ret(0);
}


texts: Reversing a linked list by iteration and recursion

texts:
From my original question on Stack Overflow: Is my implementation of reversing a linked list correct?

I'm a beginner at C and I'd like to know about style, and correctness of the reverse algorithms. What about memory management?

#include <stdio.h>
#include <stdlib.h>

typedef struct Node {
    int val;
    struct Node *next;
} Node;

Node * build_list(int num);
Node * reverse_list(Node * head);
static Node * r_reverseItem(Node * last, Node * head);
Node * r_reverse(Node * head);

/*
 * Function which iterates to 100 creating nodes
 * and linking them to head
 *
 */
Node * build_list(int num) {

    Node * head = NULL;
    Node * curr;

    int i;
    for(i=0;i<num;i++) {
    curr = (Node *)malloc(sizeof(Node));
    curr->val = i;
    curr->next = head;
    head = curr;
    }
    return head;
}

/*
 * Function which reverses linked list
 * by traversing list and flipping ptr of curr to last
 * @param head
 *
 */
Node * reverse_list(Node * head) {

    if (head == NULL) //only one elem
    return NULL;

    Node * last = NULL;
    do {
    Node * next = head->next;
    head->next = last;
    last = head;
    head = next;
    }
    while (head != NULL);
    return last;
}

/*
 * Function which recursively reverses linked list
 * @param last
 * @param head
 *
 */
static Node * r_reverseItem(Node * last, Node * head) {

    if (head == NULL)
    return last;

    Node * next = head->next;
    head->next = last;
    return r_reverseItem(head,next);
}

Node * r_reverse(Node * head) {
    return r_reverseItem(NULL, head);
}

int main (void) {
    Node * head = build_list(5);
    Node * curr;
    head = r_reverse(head);
    //    head = reverse_list(head);
    while(head) {
      printf("%d\n",head->val);
      curr = head;
      head = head->next;
      free(curr);
    }
    return 0;
}


texts: Changing all comments' size in range to autosize

texts:
I have a code that turns all comments size to autosize, however as I have quite a few cells with comments, this code runs a little slow, to the point where it will become non responsive for awhile.

Sub AutoSizeComments()
    Dim c as range, ws as worksheet

    Set ws = ActiveSheet

    'Loop through each comment on worksheet
    For Each c In ws.Cells.SpecialCells(xlCellTypeComments)

        c.Comment.Shape.TextFrame.AutoSize = True

    Next c
End Sub

After some research online, I still could not find a way to make this faster. As I am new to VBA coding, I am not sure if there is any way to make this code run faster. Thanks for helping and if this is the fastest way, let me know too. Thank you all.


texts:
The algorithm bellow is meant to allow a user to roll a die as many times as he would like, then print how many times each side was rolled and how many rolls there were total.

    import java.util.Random;
import java.util.Scanner;

public class DieRoll {

    public static void main(String[] args) {
        // TODO Auto-generated method stub

        int roll;
        int totalRolls = 0;
        int roll1 = 0;
        int roll2 = 0;
        int roll3 = 0;
        int roll4 = 0;
        int roll5 = 0;
        int roll6 = 0;
        int numberOfRolls;

        Scanner scan = new Scanner(System.in);
        Random generator = new Random();

        System.out.println("How many times would you like to roll? ");
        numberOfRolls = scan.nextInt();

    for (int numberOfLoops = numberOfRolls; numberOfLoops>0; numberOfLoops--) {

        roll = generator.nextInt(6) + 1;
        System.out.println("your roll was: " + roll);

        switch (roll) {
            case 1:
                roll1 = roll1 + 1;
                break;
            case 2:
                roll2 = roll2 + 1;
                break;
            case 3:
                roll3 = roll3 + 1;
                break;
            case 4:
                roll4 = roll4 + 1;
                break;
            case 5:
                roll5 = roll5 + 1;
                break;
            case 6:
                roll6 = roll6 + 1;
                break;
                default:
                    break;
            }
        }
        System.out.println("\nyou rolled a 1: " + roll1);
        System.out.println("you rolled a 2: " + roll2);
        System.out.println("you rolled a 3: " + roll3);
        System.out.println("you rolled a 4: " + roll4);
        System.out.println("you rolled a 5: " + roll5);
        System.out.println("you rolled a 6: " + roll6);

        System.out.println("\nYou spun a total of : " + numberOfRolls);
    }
}


texts: Detecting UI thread hanging and logging stacktrace

texts:
I have an application that needs to always be responsive. I've written a class that is designed to monitor the UI thread. The goal is to provide useful information in the logs to be able to understand when the UI thread becomes unresponsive and to determine which code is the cause.

To do this I check how long it takes to process something on the UI thread, if its longer than a certain threshold, I log a warning. If this happens multiple times in succession, it prints the stacktrace.

I've had this in production for about 2 weeks now, and so far it is working as expected.  However, given that it uses deprecated methods to get the stacktrace, I'd like to know if this might end up causing more problems than it solves.

 public class ThreadMonitor
 {
    private static readonly ILog Log =  LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
    private readonly Thread thread;
    private readonly int pollingFrequency;
    private readonly int delayThreshold;
    private readonly int stackTraceIterations;

    public ThreadMonitor(Thread thread, int pollingFrequency = 500, int delayThreshold = 100, int stackTraceIterations = 4)
    {
        this.thread = thread;
        this.pollingFrequency = pollingFrequency;
        this.delayThreshold = delayThreshold;
        this.stackTraceIterations = stackTraceIterations;
    }

    public void Run()
    {
        Task.Factory.StartNew(() =>
        {
            while (true)
            {
                Thread.Sleep(pollingFrequency);
                var dispatcher = Dispatcher.FromThread(thread);
                var task = dispatcher.InvokeAsync(() => { });

                for (var i = 0; i < stackTraceIterations; i++)
                {
                    Thread.Sleep(delayThreshold);
                    if (task.Status != DispatcherOperationStatus.Completed)
                    {
                        Log.Debug($"{(i + 1) * 100}ms Delay on thread {thread.Name} ({task.Status})");
                    }
                    else
                    {
                        break;
                    }
                }

                if (task.Status == DispatcherOperationStatus.Completed)
                    continue;

                var stackTrace = GetStackTrace(thread);
                Log.Debug($"StackTrace of UI Thread: {stackTrace}");
            }
        });
    }

    #pragma warning disable 0618
    private StackTrace GetStackTrace(Thread targetThread)
    {
        StackTrace stackTrace = null;
        var ready = new ManualResetEventSlim();

        new Thread(() =>
        {
            // Backstop to release thread in case of deadlock:
            ready.Set();
            Thread.Sleep(200);
            try { targetThread.Resume(); } catch { }
        }).Start();

        ready.Wait();
        targetThread.Suspend();
        try { stackTrace = new StackTrace(targetThread, true); }
        catch { /* Deadlock */ }
        finally
        {
            try { targetThread.Resume(); }
            catch { stackTrace = null;  /* Deadlock */  }
        }

        return stackTrace;
    }
    #pragma warning restore 0618
 }


texts: Time Converter - Python Tkinter - General comments on code

texts:
I have been learning python since being in lockdown, this is the first Tkinter application I've built without consulting the internet or seeking help anywhere.
It is a time converter where you can select the input unit via a checkbox and the other entry windows will fill with the correct conversion.
I'd like to hear feedback on how it's written, in terms of if I have demonstrated best practice or areas where I could have used less code.
from tkinter import *
from tkinter.ttk import*

root = Tk()
Second = IntVar()
Minute = IntVar()
Hour = IntVar()

Var1 = IntVar()
Var2 = IntVar()
Var3 = IntVar()

def ConvertTime():
    if Var1.get() == 1:
        Minute.set(Second.get()/60)
        Hour.set(Second.get()/3600)

    if Var2.get() == 1:
        Second.set(Minute.get()*60)
        Hour.set(Minute.get()*0.01667)

    if Var3.get() == 1:
        Second.set(Hour.get()*3600)
        Minute.set(Hour.get()*60)

class GUI:

    def __init__(self, master):

        self.master = master
        master.title('Time Converter')
        master.geometry('+600+300')

        master.bttn = Button(root, text = 'Convert', command = ConvertTime)
        master.bttn.grid(pady= 5, padx=20, column=1, row=3)

        master.ChooseSecond = Checkbutton(root, text = 'Seconds', variable=Var1, onvalue = 1, offvalue = 0)
        master.ChooseSecond.grid(pady=5, padx=20, column=0, row=0)

        master.entry1 = Entry(root, textvariable=Second)
        master.entry1.grid(pady=5, padx=20, column=0, row=1)

        master.ChooseMinute = Checkbutton(root, text = 'Minutes', variable=Var2, onvalue = 1, offvalue=0)
        master.ChooseMinute.grid(pady=5, padx=20, column=1, row=0)

        master.entry2 = Entry(root, textvariable=Minute)
        master.entry2.grid(pady=5, padx=20, column=1, row=1)

        master.ChooseHour = Checkbutton(root, text = 'Hours',variable=Var3, onvalue=1, offvalue=0)
        master.ChooseHour.grid(pady=5, padx=20, column=2, row=0)

        master.entry3 = Entry(root, textvariable=Hour)
        master.entry3.grid(pady=5, padx=20, column=2, row = 1)

def main():
    GUI(root)
    root.mainloop()

main()


texts: Interview task to efficiently find compact representation of nodes in tree and find big O time

texts:
I recently had an interview problem where I was asked to find a compact representation of nodes in a tree.

As an example, consider the following tree:

                      A
                   /  |  \
                  B   C   D
                / | \     | \
               E  F  G    H  I
              /\
              J K

The task was, given a list of node values, find a compact list of node values that completely describes the list given. A node value being in the list implictly means that all it's children are in the list as well and viceversa.

As a few examples, for the tree given:

compact([A]) = [A]
compact([B, C, D, J]) = [A]
compact([J, K]) = [E]
compact([J, K, F, G]) = [B]
compact([E, F, G, H]) = [B, H]
compact([H, I, D]) = [D]

This was my recursive attempt in python (from memory so may be some mistakes/syntax errors), which would be called with the root of the tree first. I was told:

I don't need to check for empty trees
I can assume that each node has basic tree methods (so can check it's value, if it's a leaf node etc).
The non-compact list of nodes values is passed in through node_list and I was told that there would be no duplicates in this list and that each value in it would correspond to a node in the tree

def compacted_tree(node, node_list):

    compact_list = []

    if node.value in node_list:
        #if this node is in the list then it is it's own consise representation
        return [node.value]

    allChildrenInCompactList = True

    for child in node.get_children():
        compact_list.extend(compacted_tree(child, node_list))

        if (allChildrenInCompactList):
            #only keep checking this if it's still true
            allChildrenInCompactList = child.value in compact_list

    if (allChildrenInCompactList and !node.isleaf()):
        #all my children are in the list so the compact list should just be me
        return [node.value]

    return compact_list

I said that this would run with complexity \$O(n^2)\$ (with n=number of nodes in the tree) as in the worst case, every node needs to be visited and, in the worst case, every node needs to check all it's children against the node_list which is an \$O(n)\$ operation too. I said the space complexity should be \$O(n)\$ as in the worst case you may have to add all your nodes to the compact_list before it can simplify.

It was hard to get a good read of how I went from the interview but I'm yet to get a call back. The interviewer hinted that the task could have been done with better complexity but I fail to see how. He also said that my time complexity should have been not just in terms of the number of nodes in the tree but also the length of the node_list.

Is anyone able to tell me how I should've approached this problem/analysis better?


texts: Groupby and moving average function in pandas works but is slow

texts:
I have some time series data collected for a lot of people (over 50,000) over a two year period on 1 day intervals.  I want to applying a exponential weighted moving average function for each person and each metric in the dataset.  After calculating the moving average, I want to join the new values up with the existing values in the dataframe.  I have figured out how to do this on a small sample dataset, but I'm afraid that it's not optimized and therefore won't scale to my actual dataset. I have plenty of RAM available, so I'm not afraid of running out of space, but I am concerned with how long this could take to calculate for my entire dataset. Especially as I experiment with different spans and aggregation levels.

Thanks to issue #13966 for the starting place!

How can I optimize this code to better scale to a larger dataset?

import pandas as pd
import random
from datetime import datetime

# create some sample data
vals = ['A,' * 8, 'B,' * 8, 'C,' *8]
person = []
for x in vals:
    for y in x.split(','):
        if y != '':
            person.append(y)

metric1 = [random.random() for _ in range(0, 24)]
metric2 = [random.random() for _ in range(0, 24)]
dates = [datetime(2017,1,1), datetime(2017, 2, 1), datetime(2017, 3, 1), datetime(2017, 4, 1),
         datetime(2017,5,1), datetime(2017, 6, 1), datetime(2017, 7, 1), datetime(2017, 8, 1)] * 3

# load into a dataframe
df = pd.DataFrame({'person': person,
                   'metric1': metric1,
                   'metric2': metric2,
                   'timestamp': dates})

def run_ewm(df):

    # sort the dataframe by the timestamp
    df.sort_values('timestamp', inplace=True)

    # group the df by person
    grouped = df.groupby('person')

    # create a temporary list to hold frames
    frames = []

    # iterate over the groups and apply exp. weighted moving average
    for group in grouped.groups:
        frame = grouped.get_group(group)
        frame['metric1_emw'] = frame['metric1'].ewm(span=60).mean()
        frames.append(frame)

    # concat the frames for a new dataframe
    df_new = pd.concat(frames)

    return df_new

%timeit df_new = run_ewm(df)

/home/curtis/Program_Files/miniconda2/envs/py35/lib/python3.5/site-packages/ipykernel_launcher.py:15: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  from ipykernel import kernelapp as app

10 loops, best of 3: 101 ms per loop


texts:
I'm pretty new to React and I was working on some JSON data. Now I just to understand basic concepts, so I wrote this little component, but in my eyes it's not efficient.

import React from 'react';
import {getJson} from './getJson';
import AddRow from './AddRow';

class TableComponent extends React.Component {
    constructor(props) {
      super(props)
      this.state = {
        checked: true,
        rows:[],
        json: []
      }
    }

    checkboxHandler() {}

    componentDidMount() {
      this.setState((prevState) => {
        return {
          json: getJson(),
        }
      })
    }

    checkAll = () => {
        this.setState({
            checked: !this.state.checked
        })

        this.setState(prevState => {
          const json = prevState.json.map(obj => ({
            ...obj,
            items: obj.items.map(item => ({
              ...item,
              value: this.state.checked,
            }))
          }));
          return { json };
        });
      };

    render() {
      return (

              {this.state.json.map((obj, i) => {
                return (

                    {obj.items.map((data, i) => {
                        return(

{data.label}

                                    <input
                                        type="checkbox"
                                        checked={data.value}
                                        onChange={this.checkboxHandler}
                                    />

                        )
                    })}

                )
              })}

          <button onClick={this.checkAll}>check\uncheck</button>
          <AddRow actualJson={this.state.json}/>

      )
    }
  }

  export default TableComponent;

I'm not proud of it because I feel that I'm not respecting the heart of React - use module and components. Is there a way to have cleaner code?


texts:
I have this controller that I call via Ajax, where I update the likes of a post.

I have two methods in the user model:

already_like_post?: Where I pass the posted ID and return true or false if the post is already liked.
dislike_post: Delete the user like for given post.

class LikesController < ApplicationController
    def create
        if !current_user.already_liked_post? params[:post_id]
            like = Like. new user: current_user, post_id: params[:post_id]
            if !like.save
                render json: like.errors
            end
        else
           current_user.dislike_post params[:post_id]
        end
        amount_of_likes = Post.find(params[:post_id]).likes.count
        render json:{status:"success",likes: amount_of_likes}
    end
end

These are the methods in the user model:

def already_liked_post?(post_id)
    self.likes.where(post_id: post_id).size == 1
end

def dislike_post(post_id)
    self.likes.find_by(post_id: post_id).destroy
end

Can you give a review of this? I don't know if this looks good or if it is the Rails way.


texts: Transferring a CSV file to a database while merging some columns

texts:
Say I am given a csv file that I want to import into an sqlite3 database.  The first two columns in the csv file contain unique, important information while the rest of the columns are really a list of things.  So the csv file with the header row could look like:

cucumber.csv

'important1', 'important2', 'listItem_1', 'listItem_2', 'listItem_3'
'123', '9876', 'apple', 'orange', 'banana'
'456', '7890', 'cabbage', 'onion', 'carrot'

So, when I import into the database, I want to shove all but the first two columns into one column.  The schema for the table in the database would then look like:

import csv
import sqlite3

def main():
    data_filename = 'cucumber.csv'
    db_filename = 'spam_mart.sqlite'
    SQL_create_table = """DROP TABLE IF EXISTS cucumber;
                          CREATE TABLE cucumber (
                          important1  NUMBER PRIMARY KEY,
                          important2  NUMBER,
                          item_list   TEXT
                       );
                       """
    SQL = """insert into cucumber(important1, important2, item_list)
             values (:important1, :important2, :item_list)
          """

    with open(data_filename) as f, sqlite3.connect(db_filename) as conn:
        conn.executescript(SQL_create_table)
        cursor = conn.cursor()
        reader_object = csv.reader(f)
        next(reader_object, None)      # Skip the headers
        row_count = 0
        for row in reader_object:
            row_count += 1
            row_dict = {}
            important1, important2, *item_list = row  # unpack the row
            row_dict['important1'] = important1
            row_dict['important2'] = important2
            row_dict['item_list'] = repr(item_list)  # convert list to string first
            cursor.execute(SQL, row_dict)

    print('Loaded {} of {} records'.format(str(row_count),
                                       str(reader_object.line_num - 1)))

if __name__ == '__main__':

    main()

I would normally use a csv.DictReader() object to transfer a csv file to a database, but since I was creating a list from some of the columns first, I am going with a normal csv.reader() object.

I used the repr() so that I can easily access the list again using the eval() if need be.  Script works as expected.  However, the whole technique seems a little clumsy to me.  All honest critiques welcomed.


texts: C++ object pool that returns a smart pointer on request

texts:
template<typename T>
class object_pool {

public:
  explicit object_pool(std::size_t initial_size = 1048u);

  object_pool(const object_pool&) = delete;

  object_pool(object_pool&&) = delete;

  ~object_pool();

  object_pool& operator=(const object_pool&) = delete;
  object_pool& operator=(object_pool&&) = delete;

  template<typename... Args>
  std::unique_ptr<T, std::function<void(T*)>> request(Args&&... args);

private:
  std::deque<T*> _object_pool; // is deque the best container?

};

template<typename T>
object_pool<T>::object_pool(std::size_t initial_size) : _object_pool() {
  for (std::size_t i = 0; i < initial_size; ++i) {
    T* object = reinterpret_cast<T*>(::operator new(sizeof(T)));
    _object_pool.push_back(object);
  }
}

template<typename T>
object_pool<T>::~object_pool() {
  while (!_object_pool.empty()) {
    T* object = _object_pool.front();
    _object_pool.pop_front();
    ::operator delete(object);
  }
}

template<typename T>
template<typename... Args>
std::unique_ptr<T, std::function<void(T*)>> object_pool<T>::request(Args&&... args) {
  T* object = nullptr;

  if (!_object_pool.empty()) {
    object = new(_object_pool.front()) T(std::forward<Args>(args)...);
    _object_pool.pop_front();
  } else {
    object = new T(std::forward<Args>(args)...);
  }

  auto object_pointer = std::unique_ptr<T, std::function<void(T*)>>(object, [this](T* object){
    object->~T();
    _object_pool.push_back(object);
  });

  return object_pointer;
}

My goal is to write a very lightweight object pool for another project.
The _object_pool is supposed to just hold raw uninitialized pointers to T, which I then initialize when an object is requested. Likewise, i call the destructor on the returned instances before putting them back into the _object_pool so that resources used by them can be freed.
I am also allocating space for a set ammount of instances when I create the pool and only request new memory when all object from the _object_pool are in use.


texts: Sending SMS through twilio from play framework

texts:
I am trying to send SMS using Twilio with Scala 2.10 in Playframework 2.2, but this code is looking odd to me, maybe due to lots of conditions.  I'm sure it needs some important reviews.

val invalidEX = List(21211, 21614, 21608, 30006, 30005, 21604, 21601, 14101)
val reachEX = List(21612, 30004)

def sendSms = Action { implicit request =>
    val smsTo = request.body.asFormUrlEncoded.get("number").head
    val deviceId = request.body.asFormUrlEncoded.get("deviceId").head
    try {
      if(smsTo.length<=6||smsTo.length>=18||deviceId.length>=150){
       Ok(write(Map("result" -> "error")))
      }else{
      if (smsNeeded(smsTo)) {
        if (numberBlocked(smsTo)) {
            if (!todayBlocked(smsTo)) {
              var code = random6Digit
              val appLang =request.acceptLanguages(0).code.toString.splitAt(2)._1
              var body = ""
              appLang.equals("en") match {
                case true =>
                  body = Messages("sms_body", "\n", code)(Lang(appLang))
                case false =>
                  body = Messages("sms_body", "\n", code)(Lang(appLang)) + "\n" + Messages("sms_body", "\n", code)(Lang("en"))
              }
              TwilioAPI.sendSMS(smsTo, body)
              Future {todayEntry(smsTo) }
              Ok(Json.obj("response" -> "success"))
            } else
              Ok(Json.obj("response" -> "temporaryblock"))
        } else
          Ok(Json.obj("response" -> "permanentblocked"))
      } else {
        Ok(Json.obj("response" -> "regular"))
      }
     }
    } catch {
      case twilioEx: TwilioRestException =>
        if (invalidEX.contains(twilioEx.getErrorCode))
          Ok(Json.obj("response" -> "invalidNumber"))
        else if (reachEX.contains(twilioEx.getErrorCode))
          Ok(Json.obj("response" -> "unreachableNumber"))
        else {
          Future { models.LogFile.errorLogs(Json.obj("route" -> request.tags, "post" -> request.body.asFormUrlEncoded.toString, "address" -> request.remoteAddress, "error" -> twilioEx).toString) }
          Ok(Json.obj("response" -> "error"))
        }
      case e: Exception =>
        println(e)
        Future { models.LogFile.errorLogs(Json.obj("route" -> request.tags, "post" -> request.body.asFormUrlEncoded.toString, "address" -> request.remoteAddress, "error" -> e.toString).toString) }
        Ok(Json.obj("response" -> "error"))
    }
  }


texts: Rounds off numbers from 1 - 10, 000 including negatives

texts:
I was trying to replicate Python's internal round() function.
The program asks for a number, converts it into a float (catches the value error if present) and gets the absolute value of the float (to handle negative numbers).

def rounder():
    number = input("Give me a number to round off!: \n")
    #Makes sure you enter a valid number
    try:
        floater = float(number)
        absol = abs(floater)
    except ValueError:
        print("Please enter a valid number.\n")
        starter()

    if absol < 10 and absol > 0: #Works for both 0.6 and 1.6 | Because 0.6 is too small for 1 and will return the rounding while 1.6 % == 0.6 also
        rounding = absol % 1
        if rounding >= 0.5: #If more than 0.5
            absol += 1 #Increment It
            temp =(int(absol)) #Convert back to integer

            if floater < 0: #Negative check
                print(-temp)
            else:
                print(temp)

        elif rounding < 0.5:
            temp = (int(absol))

            if floater < 0: #Negative check
                print(-temp)
            else:
                print(temp)

    elif absol < 100 and absol > 10: #use %10
        rounding = absol % 10
        rounding2 = rounding % 1
        if rounding2 >= 0.5: #If more than 0.5
            absol += 1 #Increment It
            temp = (int(absol)) #Convert back to integer

        if floater < 0: #Negative check
            print(-temp)
        else:
            print(temp)

    elif rounding2 < 0.5:
        temp = (int(absol))

        if floater < 0: #Negative check
            print(-temp)
        else:
            print(temp)
    rounder()

def starter():
    print("Compatible from 0.1 - 9999.9")
    rounder()

starter()

It keeps on going like that until if absol < 10000 and absol > 1000 . Also the number of modulo operations (%) it goes through increases.

The ranges are hard-coded so it can only handle numbers from 1 - 10 000. I could manually increase the limitation if I wanted but I would have to keep doing that until infinity.

Does anyone have any idea on how to get beyond this limitation?
Can my code be refined? Is there a more efficient way of doing this?

Keep all answers beginner-friendly.


texts: Request review of sql validation trigger

texts:
I have the following SQL structure (simplified for brevity):

TABLE [Posts]
     [Id]       INT NOT NULL IDENTITY PRIMARY KEY,
     [ParentId] INT NULL,
     [Type]     INT NOT NULL,
     FOREIGN KEY ( [ParentId] ) REFERENCES [Posts] ( [Id] )
     FOREIGN KEY ( [Type] ) REFERENCES [PostTypes] ( [Type] )

TABLE [PostTypes]
     [Id]   INT NOT NULL PRIMARY KEY,
     [Name] SYSNAME NOT NULL

TABLE [PostTypeValidParents]
     [ChildType]  INT NOT NULL,
     [ParentType] INT NOT NULL,
     PRIMARY KEY ( [ChildType], [ParentType] ),
     FOREIGN KEY ( [ChildType] ) REFERENCES [PostTypes] ( [Type] ),
     FOREIGN KEY ( [ParentType] ) REFERENCES [PostTypes] ( [Type] )

INSERT [PostTypes]
       ( [Type], [name] )
VALUES ( 0, 'Global' ),
       ( 1, 'Category' ),
       ( 2, 'Thread' ),
       ( 3, 'Reply' )

INSERT [PostTypeValidParents]
       ( [ChildType], [ParentType] )
VALUES ( 1, 0 ),
       ( 1, 1 ),
       ( 2, 1 ),
       ( 3, 2 )

Now, I want to create a trigger which validate [Posts] based on the rules defined in [PostTypeValidParents]. Here's what I have so far:

CREATE TRIGGER [ValidatePost]
ON [Posts]
AFTER INSERT, UPDATE
AS
BEGIN
    IF ( EXISTS ( SELECT * FROM [deleted] ) AND
         UPDATE ( [Type] ) )
        RAISERROR ( N'The [Posts].[Type] column cannot be modified.', 16, 1 );

    DECLARE @parents TABLE
      (
        [ChildType]  INT NOT NULL,
        [ParentType] INT NOT NULL
      );

    INSERT @parents
    SELECT c.[Type],
           CASE
             WHEN p.[Type] IS NULL THEN 0
             ELSE p.[Type]
           END
    FROM   [inserted] c
           LEFT JOIN [Forum].[Posts] p
                  ON c.[ParentId] = p.[Id];

    DECLARE @childType NVARCHAR(30),
            @parentType NVARCHAR(30);

    WITH [Conflicts] AS
      (
        SELECT [ChildType], [ParentType]
        FROM   @parents
        EXCEPT
        SELECT [ChildType], [ParentType]
        FROM   [PostTypeValidParents]
      )
    SELECT TOP 1
           @childType = c.[Name],
           @parentType = p.[Name]
    FROM   [Conflicts] x
           JOIN [PostTypes] c
             ON x.[ChildType] = c.[Type]
           JOIN [PostTypes] p
             ON x.[ParentType] = p.[Type];

    IF ( @childType IS NOT NULL )
        RAISERROR ( N'Posts of type ''%s'' may not have a parent of type ''%s''.', 16, 1, @childType, @parentType );
END

My basic pattern here is,

check for read-only fields and RAISERROR if modified
SELECT critical conditions into a table variable
find unsupported conditions with an EXCEPT inside a CTE, SELECT'ing into a variable and RAISERROR if not null.

It seems to work well enough (though I haven't thoroughly tested it yet). My problem is that this seems a bit verbose for what should be a relatively simple validation trigger. I'll have to write many of these in the near future, so I'm hoping to find a simpler pattern that accomplishes the same thing. Can anyone offer suggestions on how to reduce this code?


texts: Is there anything wrong with this unusual NSMutableArray setter?

texts:
This behaves to an outside observer mostly like @property (copy), except it has the very nice property of automatically performing any side effects that may exist in remove<Key>AtIndexes: and insert<Key>:atIndexes:. If I'm going to have such side effects, this seems really handy (DRY!)--is there anything wrong with this strange-looking accessor method?

- (void)setEdges:(NSArray *)newEdges
{
    if (newEdges != edges)
    {
        [self removeEdgesAtIndexes:[NSIndexSet indexSetWithIndexesInRange:NSMakeRange(0, [edges count])]];
        [edges release];
        edges = nil;
        if (newEdges)
        {
            edges = [NSMutableArray new];
            [self insertEdges:newEdges atIndexes:[NSIndexSet indexSetWithIndexesInRange:NSMakeRange(0, [newEdges count])]];
        }
    }
}


texts: Generating public/private RSA key pair programmatically, using it for JWTs

texts:
In my app I'm using JSON Web Tokens for authentication purposes, using the pyjwt library. Instead of using static keys and/or worrying about key distribution, the server generates a public/private key pair upon startup itself and just keeps it in memory. The keys will be reset and thereby all existing tokens invalidated when the server restarts, which is fine for the intended use case. The key pair is generated using pycrypto.

The question is: am I generating and using the keys correctly, or is there any flaw which can lead to an exploit? Below is the general class which generates and validates all JWTs in my app:

from typing import NamedTuple
from datetime import datetime
import jwt
from Crypto.PublicKey import RSA

KeyPair = NamedTuple('KeyPair', [('public', str), ('private', str)])

class JWT:
    def __init__(self, keypair: KeyPair=None, algorithm: str='RS256'):
        self.keypair = keypair or self.generate_keypair()
        self.algorithm = algorithm

    @staticmethod
    def generate_keypair() -> KeyPair:
        key = RSA.generate(2048)
        return KeyPair(public=key.publickey().exportKey('PEM').decode('ascii'),
                       private=key.exportKey('PEM').decode('ascii'))

    def generate_token(self, payload: dict) -> str:
        payload.update(dict(iat=datetime.utcnow()))
        return jwt.encode(payload, key=self.keypair.private, algorithm=self.algorithm).decode('ascii')

    def decode_payload(self, token: str, **kwargs) -> dict:
        try:
            return jwt.decode(token, self.keypair.public, algorithms=[self.algorithm], **kwargs)
        except jwt.exceptions.InvalidTokenError as e:
            raise InvalidAuthenticationToken


texts: Parsing a complicated CSV generated on election nights

texts:
I have a Python script which parses a complicated CSV generated on election nights. Each row of the CSV represents a race. As I loop through the races, I store the candidates for each race into a list called cnds. The other variable to note is called num_win, and it holds the number of people who will be elected for that particular race. Usually it's just 1, but in cases like school boards, it can be much higher.

For illustration purposes, here's some sample data we might process:

num_win = 6

cnds = [
    { 'cnd' : 'Christine Matthews', 'votes' : 200, 'winner': False },
    { 'cnd' : 'Dexter Holmes', 'votes' : 123, 'winner': False },
    { 'cnd' : 'Gerald Wheeler', 'votes' : 123, 'winner': False },
    { 'cnd' : 'Timothy Hunter', 'votes' : 100, 'winner': False },
    { 'cnd' : 'Sheila Murray', 'votes' : 94, 'winner': False },
    { 'cnd' : 'Elisa Banks', 'votes' : 88, 'winner': False },
    { 'cnd' : 'John Park', 'votes' : 88, 'winner': False },
    { 'cnd' : 'Guadalupe Bates', 'votes' : 76, 'winner': False },
    { 'cnd' : 'Lynne Austin', 'votes' : 66, 'winner': False }
]

First attempt:

My initial version was pretty straightforward. Make a copy of cnds, sort it in order of vote count, and limit to all but the num_win number of candidates. These are the winners. Then loop through cnds and mark the winners.

winners = sorted(cnds, key=lambda k: int(k['votes']), reverse=True)[0:num_win]
for cnd in cnds:
    for winner in winners:
        if cnd['cnd'] == winner['cnd']:
            cnd['winner'] = True

This works great -- except I realized later that it doesn't account for ties.

Since this script is for election night when the results are unofficial, I only want to mark as winners the candidates I am sure are winners. In the data above, the clear winners would be: Christine Matthews, Dexter Holmes, Gerald Wheeler, Timothy Hunter, and Sheila Murray. There is a tie for the sixth spot. Depending on the type of race, etc, it might be settled later by a runoff or some other mechanism. So, on election night, I simply wouldn't mark anyone else after those 5 as being a winner.

Here's the new code I've written, which accounts for tie situations:

# Make list of unique vote totals, with number of candidates who had those vote totals
# This code uses collections.Counter to make the list of uniques.
# http://stackoverflow.com/a/15816111/566307
uniques = Counter(cnd['votes'] for cnd in cnds).iteritems()

# Now convert the Counter() output into a sorted list of tuples.
uniquesCount = sorted( uniques, reverse=True )[0:num_win]

# How many candidates are there in this list?
# http://stackoverflow.com/a/14180875/566307
cndsInUniques = map(sum,zip(*uniquesCount))[1]

# There's too many candidates. Must be one or more ties
if cndsInUniques > num_win:
    adjusted_num_win = num_win

    # We need to remove items from the uniques list until we get the
    # num of candidates below or equal to the num_win threshold.
    while len(uniquesCount) > 0:
        # delete last item
        del uniquesCount[-1]
        cndsInUniques = map(sum,zip(*uniquesCount))[1]
        if cndsInUniques <= num_win:
            adjusted_num_win = cndsInUniques
            break
    winners = sorted(cnds, key=lambda k: int(k['votes']), reverse=True)[0:adjusted_num_win]

# Right number of candidates means no ties. Proceed as normal.
else:
    # Make list of candidates, sorted by vote totals
    winners = sorted(cnds, key=lambda k: int(k['votes']), reverse=True)[0:num_win]

# loop through all candidates and mark the ones who are winners
for cnd in cnds:
    for winner in winners:
        if cnd['cnd'] == winner['cnd']:
            cnd['winner'] = True

This code is working for me, but I feel like it's a lot of work to reach the adjusted_num_win number that I need. Can anyone suggest an alternative, or ways I might simplify this?


texts:
I wrote this class which reads an SSH config file and checks whether an IP exists. If the IP does not exist, the class adds a block for that IP address.

Header file:

class SSHConfigFile : public QObject
{
    Q_OBJECT
public:
    explicit SSHConfigFile(QObject *parent = 0);

signals:

public slots:

    void registerIP(QString ip);

private:
    void read();
    bool find(QString ip);
    void addIP(QString ip);
    void write();

private:
    // the file name with path of config file
    QString _sshConfigFileName;

    QFile file;

    // the actual file data
    QString fileData;

    // the same file data but split into lines
    QStringList fileLines;

};

Implementation file:

SSHConfigFile::SSHConfigFile(QObject *parent) : QObject(parent),
    _sshConfigFileName( QStandardPaths::writableLocation( QStandardPaths::HomeLocation ) + "/" + ".ssh/config"),
    file( _sshConfigFileName )
{

}

void SSHConfigFile::read()
{
   if ( !file.open( QFile::ReadWrite | QFile::Text) )
   {
       qDebug() << "Could not open file for writing";
       return;
   }

   // read all data
   QTextStream in(&file);
   fileData = in.readAll();
   qDebug() << fileData;

   fileLines = fileData.split(QRegExp("[\r\n]"),QString::SkipEmptyParts);
}

// returns true if ip was found in config file otherwise false
bool SSHConfigFile::find(QString ip)
{
    read();

    foreach(QString  str, fileLines )
    {
        str = str.trimmed();

        //qDebug() << str;

        if (str.contains( "Host", Qt::CaseInsensitive))
        {
            str.remove("Host", Qt::CaseInsensitive);
            str = str.trimmed();

            if (str == ip)
                return true; // found
        }
    }

    return false; // not found
}

void SSHConfigFile::registerIP(QString ip)
{
    if (!find( ip))
    {
        // this is new ip, add it to config file
        addIP( ip );
    }
}

void SSHConfigFile::addIP(QString ip)
{
    QString addendum = QString("# Added by RigakuSync\n"
                               "Host %1\n"
                               "\tStrictHostKeyChecking no\n"
                               "\tUserKnownHostsFile=/dev/null\n").arg(ip);

    fileData = "\n" + addendum;

    write();
}

void SSHConfigFile::write()
{
    QTextStream out(&file);
    out << fileData;
    file.flush();
    file.close();
}

Sample use for additional context:

void DeviceSync::syncDevice(QString ip)
{
    // register IP to ssh config file if not already there.
    // I chose to do it on in function than to first check here if
    // this ip exist and if not than add it..fair enough?
    SSHConfigFile().registerIP( ip );

    // and then the rest of the code to perform sync. Basically i need to
    // make sure the ip is there in config file before i perform sync
}

From the code, I only call the registerIP() function, which is the only public function. It automatically adds the IP if it's not already in the config file.

When I pass the IP address, should the class just store it internally and then work with it, or is passing in between member functions better? Also, do the various member functions make sense or could it all be done in just one function?

Is writing a class for this a good approach or can this whole thing just be a function in another class? If this class is a good approach, can it be improved?


texts:
I have recently started looking into networking with Python. I saw some tutorials about the socket library so I decided to give a "live chat room" idea a go. It works (on Windows, os.system("cls") may be problematic) but was just wanting to see how people could improve upon it

server.py

from socketserver import ThreadingTCPServer, BaseRequestHandler
from threading import Thread
import pickle,time,datetime,os

messages = []
temp = []
os.system("cls")
class Echo(BaseRequestHandler):

    def handle(self):
        self.temp = []
        Thread(target=self.send).start()
        self.username = self.request.recv(8192)
        self.username = self.username.decode()
        print("Got connection from {}:{}".format(self.client_address[0],
                                                 self.client_address[1]))

        while True:
            msg = self.request.recv(8192)
            msg = "[{} {}]: {}".format(datetime.datetime.now().strftime("%H:%M:%S"),
                               self.username,
                               msg.decode())

            messages.append(msg)
            print(msg)

            if not msg:
                break

    def send(self):

        global temp, messages
        while 1:

            if len(self.temp) != len(messages):

                data_string = pickle.dumps(messages)
                self.request.send(data_string)
                self.temp = [item for item in messages]
if __name__ == "__main__":
    serv = ThreadingTCPServer(("",20000), Echo)
    serv.serve_forever()

client.py

import socket,pickle,os
from threading import Thread
import time
s = socket.socket()
s.connect(('localhost',20000))
def receive():

    while True:

        data = s.recv(8192)
        data = pickle.loads(data)
        os.system("cls")
       for item in data:
            print(item)

username = input("Enter your username: ")
Thread(target=receive).start()

s.send(username.encode())
time.sleep(0.1)
while True:

    msg = input(">>")

    if not msg:
       break
    s.send(msg.encode())
    time.sleep(0.1)


texts: Verifier for multi-purpose scripting language

texts:
I rewrote a compiler project right now containing semantics, parser and verifier, in Java. Links: source code, language overview. The language is familiar for ECMAScript.

As an example, the verifier explicitly marks 'frame properties' (or locals) captured by inner functions and also marks 'bound methods' (indirectly-called instance methods):

public Symbol resolveLexicalProperty(ast.SimpleIDNode id, int flags, boolean reportError)
{
    // ...

    if (p.isObjectPropertyAccess())
    {
        if (p.accessingProperty().isFunction() && (flags & VerifyFlags.OBJECT_CALL) == 0)
        {
            p.accessingProperty().setIsBoundMethod(true);
        }
        if (p.baseObject().isThis())
        {
            if (p.baseObject().activation() != currentActivation)
            {
                p.baseObject().activation().setScopeExtendedProperty(p.baseObject());
            }
        }
        // code may capture property from an enclosing with frame, such as in:
        //
        // with (0x0a)
        // {
        //     function f():void { trace(toString(16)) }
        // }
        //
        else if (p.baseObject().isFramePropertyAccess())
        {
            var p2 = p.baseObject();
            if (p2.baseFrame().activation() != this.currentActivation)
            {
                p2.baseFrame().activation().setScopeExtendedProperty(p2.accessingProperty());
            }
        }
    }
    else if (p.isUnionPropertyAccess() && p.accessingProperties().get(0).isFunction() && (flags & VerifyFlags.OBJECT_CALL) == 0)
    {
        for (var p2 : p.accessingProperties())
        {
            p2.setIsBoundMethod(true);
        }
    }
    else if (p.isFramePropertyAccess())
    {
        if (p.baseFrame().activation() != currentActivation)
        {
            p.baseFrame().activation().setScopeExtendedProperty(p.accessingProperty());
        }
    }

    // ...
}

This is part of the org.shockscript.sxc.verifier.Verifier class. The local p above  is a org.shockscript.sxc.semantics.Symbol object (this class defines various abstract methods).

About the parser, the nodes are all defined in the static class org.shockscript.sxc.parser.ast. Some of the nodes contain Symbol references.

Is this good for codegen?

To build the project, Maven and JDK 14+ are required (openjdk-14-jdk), using the commands mvn compile and . run.sh arguments in the project root. To distribute JAR, mvn package.

Also, when there is no bytecode to import, the compiler auto loads the language built-ins. There are various sources being loaded in the builtins directory of the compiler repo.


texts: Hacker Rank: Extracting digits from a given number and check for divisibility

texts:
The problem in question is a coding challenge from Hackerrank:

  Problem Statement

  You are given an integer N. Find the digits in this number that
  exactly divide N (division that leaves 0 as remainder) and display
  their count. For N=24, there are 2 digits (2 & 4). Both of these
  digits exactly divide 24. So our answer is 2.

  Note

  If the same number is repeated twice at different positions, it should
  be counted twice, e.g., For N=122, 2 divides 122 exactly and occurs at
  ones' and tens' position. So for this case, our answer is 3. Division
  by 0 is undefined. Input Format

  The first line contains T (the number of test cases), followed by T
  lines (each containing an integer N).

  Constraints
  1≤T≤15
   0 < N <10 ^ 10

  Output Format

  For each test case, display the count of digits in N that exactly
  divide N in a separate line.

Here's my code:

package algorithms.Warmup;

import java.util.ArrayList;
import java.util.Scanner;

/**
 * Created by user1 on 4/2/15.
 */
public class FindDigitsTest {

    static Scanner sc = new Scanner(System.in);

    private static Long[] findDigits(long num) {

        ArrayList<Long> digits = new ArrayList<>();

        //extract all digits from input number and store in arraylist
        while (num > 0) {

            long quotient = num / 10;
            long remainder = num - (quotient * 10);
            num = quotient;
            digits.add(remainder);

        }

        Long[] result = new Long[digits.size()];
        result = digits.toArray(result) ;
        return result;

    }

    private static int findCount(Long[] list, long num) {

        int count = 0;
        for(int i = 0; i < list.length; i++) {

            //ignore division by zero- DON'T FORGET!!!
            if(list[i] == 0) {
                continue;
            }
            if (num % list[i] == 0){
                count++;
            }
        }

        return count;

    }
    public static void main(String[] args) {

        ArrayList<Integer> al = new ArrayList<>();

        //read in number of test cases
        System.out.println("Enter number of test cases: ");
        int T = sc.nextInt();

        //loop through number of test cases
        for(int i = 0; i < T; i++) {

            System.out.println("Enter the number: ");
            long N = sc.nextLong();

            //extract digits of number and store in array
            Long[] nDigits = findDigits(N);

            //calculate num. of digits which exactly divide N
            int answer = findCount(nDigits, N);

            //store answer in final arraylist
            al.add(answer);

        }

        //loop through final arraylist to print answers
        for(int i = 0; i < al.size(); i++) {
            System.out.println(al.get(i));
        }

    }
}

Even though the submission is correct, and the output is as desired, I feel there are many ways this can be improved, I just don't know how. I see that I have too many ArrayLists and Arrays here and there throughout the code, maybe that can be replaced to begin with.

I'm not completely proficient in Java yet, so my knowledge of Java APIs and Collections is still somewhat limited. That might be one reason for the inefficient code.


texts: Loop through List multi-threading to ping hostnames

texts:
I am new to Multi-threading. I have had a go at writing some code that loops through an ArrayList, gets a hostname and then pings the hostname. If the hostname is not pingable it adds the hostname to a list returned by the method.

I am very new to multi-threading so I was just wondering if someone could give me a hand with improving this. For example I've heard that you can pass an ArrayList to Runnable but I can't find any good examples on how to do this yet.

Please note that I am using Java 5.

private ArrayList<String> pingNonReplicatingRegisters (
    Collection<RegisterReplicationSynchTime> nonReplicatingRegisters) throws IOException {

    final ArrayList<String> nonPingableRegisters = new ArrayList<String>();

    int nThreads = 15;
    final ExecutorService es = Executors.newFixedThreadPool(nThreads);

    for(RegisterReplicationSynchTime nonReplicatingRegister : nonReplicatingRegisters){

        final String registerHostName = nonReplicatingRegister.getRegisterName();

        es.execute(new Runnable(){
            public void run(){
                boolean status = false;
                Socket socket = null;
                PingBean pngBean = new PingBean();

                try {
                    SocketAddress socketAddress = new InetSocketAddress(registerHostName, 139);
                    socket = new Socket();
                    socket.connect(socketAddress, 1000); //1secs
                    status = true;
                    socket.close();
                } catch (IOException ex) {

                }
                finally {
                    if (socket != null) {
                        try {
                            socket.close();
                        } catch (PortUnreachableException e) {
                            socket = null;
                            System.err.println("Got an " + e);
                        } catch (UnknownHostException e) {
                            // TODO Auto-generated catch block
                            socket = null;
                            System.err.println("Got an " + e);
                        } catch (IOException e) {
                            socket = null;
                        }
                    }
               }// finally
               /* return status */
               if (!status) {
                   nonPingableRegisters.add(registerHostName);
               }
           }
        });
   }// for
   try {
       es.shutdown();
       es.awaitTermination(60, TimeUnit.SECONDS);
       List<Runnable> pending = es.shutdownNow();

       if (!pending.isEmpty()) {
           System.err.println("Pending threads: " + pending);
       }

    } catch (InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    return nonPingableRegisters;
}


texts:
Create a binary search tree representation of a given array of integers.

How can I improve this code?

# Creating Node object class
class Node(object):
    def __init__(self, value, left=None, right=None):
        self.value = value
        self.left = left
        self.right = right

# Creating Tree class
class BST(object):
    def __init__(self, val):
        self.root = Node(val)

    def insert(self, value):
        current = self.root
        while current:
            if value < current.value:
                if current.left == None:
                    current.left = Node(value)
                    break
                else:
                    current = current.left
            else:
                if current.right == None:
                    current.right = Node(value)
                    break
                else:
                    current = current.right

# inputs
t = BST(22)
t.insert(30)
t.insert(10)
t.insert(80)
t.insert(90)
t.insert(9)
t.insert(23)
t.insert(14)
t.insert(6)
t.insert(40)
t.insert(60)
t.insert(3)


texts: Reading and writing of pixels of BufferedImages

texts:
I'm working on a project that takes in a video file and does some image processing on each individual frame. The first step in that processing is to take each source frame and generate three monochromatic images based upon the average red, green, and blue values of the image. If that doesn't make sense, I've drawn up an example image for illustration:

I achieve this step by passing in a BufferedImage (along with a frame number) to a function that loops over the individual pixels and extracts the color data (via getRGB()), and then setRGB()s it into the the three new images as is appropriate.

Unfortunately the performance is killing me, with it taking up to 4 seconds to render the three images per frame (and, by the way, this is with test footage whose resolution is only 856x480! I'll need to be able to use it on footage of 1080p!) For a 30 second clip at 30FPS, this first step takes around an hour! (~4 seconds x 30 seconds x 30 frames = 3600 seconds) I admit, my 1.4GHz dual core really isn't cut out for this kind of stuff, but if I can render 4 minutes of 1080p 60FPS footage in 10 minutes using Virtualdub, then I know I can squeeze more performance out of this thing.

The code I wrote is below. I'm using Rhino to script Java, but I think it should be clear even to those who normally don't use JavaScript.

importPackage(java.io)
importPackage(javax.imageio)
importPackage(java.awt.image)
importPackage(java.awt)

function makeMonochromaticImages(img, n) {
  var w = img.getWidth()
  var h = img.getHeight()

  // These are the three monochromatic images produced.
  var red   = new BufferedImage(w, h, BufferedImage.TYPE_INT_RGB);
  var green = new BufferedImage(w, h, BufferedImage.TYPE_INT_RGB);
  var blue  = new BufferedImage(w, h, BufferedImage.TYPE_INT_RGB);

  // I've cached the constructor to avoid constant overload resoloution
  var _Color = Color['(int,int,int)']

  for(var x = 0; x < w; ++x) {
    for(var y = 0; y < h; ++y) {
      var c = new Color(img.getRGB(x, y))
      var r = c.getRed()
      var g = c.getGreen()
      var b = c.getBlue()

      var red_grayscaled   = (new _Color(r, r, r)).getRGB()
      var green_grayscaled = (new _Color(g, g, g)).getRGB()
      var blue_grayscaled  = (new _Color(b, b, b)).getRGB()

      red.setRGB(x, y, red_grayscaled)
      green.setRGB(x, y, green_grayscaled)
      blue.setRGB(x, y, blue_grayscaled)
    }
  }

  ImageIO.write(red, "png", new File("work\\newframes\\" + (n) + "-r.png"))
  ImageIO.write(blue, "png", new File("work\\newframes\\" + (n) + "-g.png"))
  ImageIO.write(green, "png", new File("work\\newframes\\" + (n) + "-b.png"))
}

I've already considered some other potential options such as manually converting between Integer and RBG, but this seems really messy. I could also make it multi-threaded and render twice as much at once and fully utilize my two cores, but then the code is still technically slow. I guess I'm looking more for a big picture idea (but I'm also not shying away from micro-optimizations if they will have major impact on performance. Remember for a 856x480 image 410880 pixels have to be processed, so I'm actually creating a lot of Color objects.)


texts: Looping through columns in Entity Framework

texts:
Is there a cleaner/more efficient way to loop through the columns in EF implicitly than the way written below?

static void Main(string[] args) {
    using (var db = new someDbContext()) {
        var query = from p in db.someTable
                    select new {
                        column1 = p.column1
                        column2 = p.column2
                    };

        var columnAccessors = CreateAccessors(query.FirstOrDefault());

        foreach (var row in query) {
            foreach (var col in columnAccessors) {
                var val = col(row);
                //Do something with val here.
            }
        }
    }
}

static Func<T, object>[] CreateAccessors<T>(T source = default(T)) {
    var propertyAccessors = typeof(T).GetProperties(BindingFlags.Instance | BindingFlags.Public)
                .Where(p => p.CanRead)
                .Select((p, i) => new {
                    Index = i,
                    Property = p,
                    Accessor = CreatePropertyAccessor<T>(p)
                })
                .ToArray();

    return propertyAccessors.Select(p => p.Accessor).ToArray();
}

static Func<T, object> CreatePropertyAccessor<T>(PropertyInfo prop) {
    var param = Expression.Parameter(typeof(T), "input");
    var propertyAccess = Expression.Property(param, prop.GetGetMethod());
    var castAsObject = Expression.TypeAs(propertyAccess, typeof(object));
    var lambda = Expression.Lambda<Func<T, object>>(castAsObject, param);
    return lambda.Compile();
}

I've tried simply doing a foreach (var col in row) before but I know that it won't work because row doesn't contain a definition for GetEnumerator and I'm not sure how one would go about implementing a solution that would do so generically for something like this.

I was working on implementing a DataReader and came across this page: http://www.developerfusion.com/article/122498/using-sqlbulkcopy-for-high-performance-inserts/. The thought occurred to me that I could modify the CreatePropertyAccessors segments to loop through columns and so I came up with the above. Problem being I'm just not sure that it's a "good" solution.

Edit:

With one small change the following is possible as well:

var query = from p in db.someTable
            select new SomeModel {
                item1 = p.column1,
                item2 = p.column2
            };

var columnAccessors = CreateAccessors<SomeModel>();


texts: Optimize function that does phrase-search and lists them

texts:
I am writing a long piece of code, which is taking way too long to execute. I used  cProfile on the code, I found that the following function is called 150 times and takes 1.3 seconds per call, leading to around 200 seconds to this function alone. The function is -

def makeGsList(sentences,org):
    gs_list1=[]
    gs_list2=[]
    for s in sentences:
        if s.startswith(tuple(StartWords)):
            s = s.lower()
            if org=='m':
                gs_list1 = [k for k in m_phrases if k in s]
            if org=='h':
                gs_list1 = [k for k in h_phrases if k in s]
            for gs_element in gs_list1:
                gs_list2.append(gs_element)
    gs_list3 = list(set(gs_list2))
    return gs_list3

The code is supposed to take a list of sentences and a flag org. Then it goes through each line, checks if it starts with any of the words present in the list StartWords, and then lower-cases it. Then, depending on the value of org, it makes a list of all phrases in the current sentence which are also present in either m_phrases or h_phrases. It keeps appending these phrases to another list gs_list2. Finally it makes a set of gs_list2 and returns it.

Can someone give me any suggestion as to how I can optimize this function to reduce the time taken to execute?

Some examples -

StartWords = ['!Series_title','!Series_summary','!Series_overall_design','!Sample_title','!Sample_source_name_ch1','!Sample_characteristics_ch1']

sentences = [u'!Series_title\t"Transcript profiles of DCs of PLOSL patients show abnormalities in pathways of actin bundling and immune response"\n', u'!Series_summary\t"This study was aimed to identify pathways associated with loss-of-function of the DAP12/TREM2 receptor complex and thus gain insight into pathogenesis of PLOSL (polycystic lipomembranous osteodysplasia with sclerosing leukoencephalopathy). Transcript profiles of PLOSL patients\' DCs showed differential expression of genes involved in actin bundling and immune response, but also for the stability of myelin and bone remodeling."\n', u'!Series_summary\t"Keywords: PLOSL patient samples vs. control samples"\n', u'!Series_overall_design\t"Transcript profiles of in vitro differentiated DCs of three controls and five PLOSL patients were analyzed."\n', u'!Series_type\t"Expression profiling by array"\n', u'!Sample_title\t"potilas_DC_A"\t"potilas_DC_B"\t"potilas_DC_C"\t"kontrolli_DC_A"\t"kontrolli_DC_C"\t"kontrolli_DC_D"\t"potilas_DC_E"\t"potilas_DC_D"\n',  u'!Sample_characteristics_ch1\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\n', u'!Sample_description\t"DAP12mut"\t"DAP12mut"\t"DAP12mut"\t"control"\t"control"\t"control"\t"TREM2mut"\t"TREM2mut"\n']

h_phrases = ['pp1665', 'glycerophosphodiester phosphodiesterase domain containing 5', 'gde2', 'PLOSL patients', 'actin bundling', 'glycerophosphodiester phosphodiesterase 2', 'glycerophosphodiester phosphodiesterase domain-containing protein 5']

m_phrases are similar. Assume in this case, org=h.

Regarding sizes -

The length of both lists h_phrases and m_phrases is around 250,000. And each element in the lists is on an average 2 words long. The list of sentences is around 10-20 sentences long and I have provided an example list to give you the idea of how big each sentence can be.


texts: 99 bottles of beer on the wall + EcmaScript 6

texts:
Today, a solution written in Python was posted to this challenge, and since I was inspired by it, I made a version in EcmaScript 6.

I tried to explore some new language constructs:

arrow functions
template Strings
let
for..of
generators

The code:

function* bottlesOfBeer(howMany=99){
  let pluralizeBottles = count => count !== 1 ? "bottles" : "bottle";
  let takeNext = next => next >= 1 ? next.toString() : "No more";
  for (let i = howMany; i >= 0; i--) {
    let bottle = pluralizeBottles(i);
    let bottleNext = pluralizeBottles(i-1);
    let left = takeNext(i);
    let toTake = i > 1 ? "one" : (i == 1 ? "it" : "no more");
    let toTakeNext = takeNext(i-1);
    let line1 = `${left} ${bottle} of beer on the wall, ${left.toLowerCase()} ${bottle} of beer.`;
    let line2 = `\nTake ${toTake} down and pass it around, ${toTakeNext.toLowerCase()} ${bottleNext} of beer on the wall.\n`;
    yield `${line1}${i > 0 ? line2 : ""}`;
  }
  yield `Go to the store and buy some more, ${howMany} bottles of beer on the wall.`;
}

for (let it of bottlesOfBeer()) {
  console.log(it);
}

This code can be copy/pasted into here to see it working: http://6to5.org/repl/

Did I use these constructs appropriately? I'd like to know if I somehow abused them.

And could it be made more readable?

What do you think, overall?


texts:
I have implemented a trie data structure in my way, though it is very similar to what major implementations I found but would like to get it reviewed by experts.

I don't see the need to store character in TrieContainer class as position within the series reflects the character stored. Maybe I am missing something and am wrong in my understanding.

Trie container

package trie;

public class TrieContainer{
    public boolean isEnd;
    public TrieContainer[] series = new TrieContainer[26];
}

Operations class

package trie;

public class Trie {

    public static void main(String[] args) {
        TrieContainer start = new TrieContainer();

        storeWords(start, "abc");
        storeWords(start, "abcd");
        storeWords(start, "abcde");
        storeWords(start, "india");
        storeWords(start, "hello");
        storeWords(start, "hell");

        printWordStrings(start,"");

        System.out.println("\n"+isWordPresent(start, "abcd"));
    }

    private static void storeWords(TrieContainer start, String word){
        for (int j = 0; j < word.length(); j++) {
            char character = word.charAt(j);

            //In series, check the position of character,
            //if it is already filled then check the series of filled Trie object.
            //if it is not filled then create new TrieContainer object and place it at correct position, and check
            //if it is end of the word, then mark isEnd = true or else false;
            if(start.series[character-97]!=null){
                if(word.length()-1 == j){ //if word is found till last character, then mark the end as true.
                    start.series[character-97].isEnd = true;
                }
                start = start.series[character-97];
            }else{
                TrieContainer trie = new TrieContainer();
                trie.isEnd = (word.length()-1 == j ? true:false);
                start.series[character-97] = trie;
                start = start.series[character-97];
            }
        }
    }

    private static boolean isWordPresent(TrieContainer start, String word){
        boolean isFound = true;
        for (int i = 0; i < word.length(); i++) {
            char character = word.charAt(i);

            //if at character position TrieContainer object is present then character is found and
            //start looking for next character is word.
            if(start.series[character-97]!=null){
                if(word.length()-1 != i){
                    start = start.series[character-97];
                }else{
                    if(!start.series[character-97].isEnd){
                        isFound = false;
                    }
                }
            }else{
                isFound = false;
                break;
            }
        }
        return isFound;
    }

    private static void printWordStrings(TrieContainer start, String toPrint) {
        if(start==null){
            return;
        }
        if(start.isEnd){
            System.out.println(toPrint);
        }

        for (int i = 0; i < start.series.length; i++) {
            TrieContainer t = start.series[i];
            if(t!=null){
                printWordStrings(t, toPrint + (char)(97+i));
            }
        }
    }
}


texts: Recursive function that generates the permutations of a string

texts:
I am looking for a review of my recursive function that generates the permutations of a string. Are there better ways to do this?

var permutations = [];

function doPerm(str, arr) {
    if (typeof (str) == 'string') str = str.split('');
    if (str.length == 0) permutations.push(arr.join(''));
    for (var i = 0; i < str.length; i++) {
        var x = str.splice(i, 1);
        arr.push(x);
        doPerm(str, arr);
        arr.pop();
        str.splice(i, 0, x);
    }
}
doPerm('', []);
console.log(permutations);


texts: Insert items from one table that do no exist in the other one

texts:
The insert query below must insert the items not present in the Articolo table by importing them from Importazione, and must update the items in the Articolo with those present in the Importazione. To summarize in Articolo the articles must be updated and the new ones included. The fact is that the two queries used by c # are very slow. How can I speed them up?

Sql Execution Plan Insert : https://www.brentozar.com/pastetheplan/?id=ryifseaOX

C# Code:

String QueryInserimentoNuoviArticoli = "Insert into Articolo(CodArt,Descrizione,CodMarca,CodEAN,Prezzo,PrezzoListino,UM,Fornitore,Importato) SELECT ArticoloMetel as CodArt,DescrizioneArticolo as Descrizione,MarcaMetel as CodMarca,CodiceBarreMetel as CodEAN,PrezzoNetto,PrezzoListino,UM,MarcaMetel as Fornitore,'ELETTROVENETA' as Importato FROM Importazione where ArticoloMetel not in ( select CodArt from Articolo where Importato = 'ELETTROVENETA' ) and MarcaMetel not in ( select CodMarca from Articolo where Importato = 'ELETTROVENETA' ) ";
SqlCommand command2 = new SqlCommand(QueryInserimentoNuoviArticoli, conn)
{
  CommandTimeout = 0
};
command2.ExecuteNonQuery();

Extracted queries

Insert Query:

INSERT INTO Articolo(CodArt, Descrizione, CodMarca, CodEAN, Prezzo, PrezzoListino, UM, Fornitore, Importato)
SELECT ArticoloMetel AS CodArt,
       DescrizioneArticolo AS Descrizione,
       MarcaMetel AS CodMarca,
       CodiceBarreMetel AS CodEAN,
       PrezzoNetto,
       PrezzoListino,
       UM,
       MarcaMetel AS Fornitore,
       'ELETTROVENETA' AS Importato
FROM Importazione
WHERE ArticoloMetel NOT IN
    (SELECT CodArt
     FROM Articolo
     WHERE Importato = 'ELETTROVENETA' )
  AND MarcaMetel NOT IN
    (SELECT CodMarca
     FROM Articolo
     WHERE Importato = 'ELETTROVENETA' )


texts: Setting a default value if a variable is empty

texts:
This is the pattern I've been using for "if the variable is empty, set a default value", and for variables like $muffin, it has all seemed well and good.  But in the following real example, this pattern gives me a super long line which smells a bit to me.  Is there a cleaner way to do this?

$newsItems[0]['image_url']= $newsItems[0]['image_url']=='' ? '/img/cat_placeholder.jpg' : $newsItems[0]['image_url'];

Maybe:

$newsItems[0]['image_url']= $newsItems[0]['image_url']=='' ?
    '/img/cat_placeholder.jpg' :
    $newsItems[0]['image_url'];   //this doesn't look much better to me?


texts: change space to underscore in filename using C

texts:
I have written my first C program that renames spaces in filenames to underscores. I wonder if there is an easier or more efficient way:

#include <stdio.h>
#include <string.h>

int main( int argc, char *argv[] ) {

    if ( argc != 2 ) {
        printf("Format is: NoSpace <filename>\n");
        return(0);
    }
    else if ( strstr( argv[1], " ") == NULL) {
        printf("No space character in file: %s\n",argv[1]);
        return(0);
    }

    int filename_size = strlen(argv[1]);
    char file_name_old[filename_size], file_name_new[filename_size];

    strcpy(file_name_old, argv[1]);
    strcpy(file_name_new, argv[1]);

    // ASCII 32 = space; ACII 95 = underscore
    for( int pos = 0; pos < filename_size; pos++) {
        if ( file_name_new[pos] == 32)  {
            file_name_new[pos] = 95;
        }
    }

    rename(file_name_old,file_name_new);
    return(0);
}

Thanks!


texts:
As an exercise, I wanted to make an array implementation of Stack, including most of the methods from the native Java Stack.

I'd like to know the following:

Does the code follow good programming principles?
Is the code readable/understandable?
Are there any redundant fields/operations?
Is there a way to further improve the code?
Should I reduce the capacity of the stack at some point? Testing the native Java Stack, it doesn't seem to do any size reduction, except with trim().

Note: I know Javadoc is missing, but I am only just beginning to read into that.

import java.util.Arrays;
import java.util.EmptyStackException;
import java.util.Random;

@SuppressWarnings("unchecked")
public class ArrayStack<T> {
    private T[] data;
    private int numElements;
    private static final int INITIAL_CAPACITY = 10;

    public ArrayStack() {
        data = (T[]) new Object[INITIAL_CAPACITY];
    }

    private void enlarge() {
        T[] temp = Arrays.copyOf(data, data.length);
        data = (T[]) new Object[data.length * 2];

        for (int i = 0; i < temp.length; i++) {
            data[i] = temp[i];
        }
    }

    public void push(T element) {
        if (numElements == data.length) {
            enlarge();
        }

        data[numElements] = element;
        numElements++;
    }

    public T peek() {
        if (isEmpty()) throw new EmptyStackException();
        return data[numElements - 1];
    }

    public T pop() {
        if (isEmpty()) throw new EmptyStackException();
        T element = data[--numElements];
        data[numElements] = null;
        return element;
    }

    public boolean isEmpty() {
        return (numElements == 0);
    }

    public int capacity() {
        return data.length;
    }

    public int size() {
        return numElements;
    }

    public void clear() {
        for (int i = 0; i < numElements; i++) {
            data[i] = null;
        }

        numElements = 0;
    }
}


texts: Create a Tree Node from JavaScript Array

texts:
I'm trying to create a list of subpaths from a flattened array.

Here's the starting point:

var flat = [
  {path: "/"  },
  {path: "A"  },
  {path: "A.A"},
  {path: "A.B"},
  {path: "A.C"},
  {path: "B"  },
  {path: "C"  }
]

The goal is that anything under a parent would be a subpath of that.  So the goal would look like this:

var goal = [
  {path: "/"  },
  {path: "A",
   subpaths: [
    {path: "A.A"},
    {path: "A.B"},
    {path: "A.C"}
   ]
  },
  {path: "B"  },
  {path: "C"  }
]

I've hobbled together a solution, but it doesn't feel very clean and is probably prone to breakage:

var nested = [];
for (var i=0; i < flat.length; i++) {
  // split components up
  var routes = flat[i].path.split(".");

  // get parent key
  var key = routes[0];

  // check if we've already added the key
  var index = -1;
  for (var j=0; j < nested.length; j++) {
    if (nested[j].path == key) {
      index = j;
      break;
    }
  }

  if (index===-1) {
    // if we have a new parent add it
    nested.push(flat[i])

  } else {
    // create subpaths property on new object
    if (!nested[index].subpaths) { nested[index].subpaths = [] }

    // add child paths to existing parent
    nested[index].subpaths.push(flat[i])
  }
}

Here's a working Demo in Stack Snippets

var flat = [
  {path: "/"  },
  {path: "A"  },
  {path: "A.A"},
  {path: "A.B"},
  {path: "A.C"},
  {path: "B"  },
  {path: "C"  }
]

var nested = [];
for (var i=0; i < flat.length; i++) {
  // split components up
  var routes = flat[i].path.split(".");

  // get parent key
  var key = routes[0];

  // check if we've already added the key
  var index = -1;
  for (var j=0; j < nested.length; j++) {
    if (nested[j].path == key) {
      index = j;
      break;
    }
  }

  if (index===-1) {
    // if we have a new parent add it
    nested.push(flat[i])

  } else {
    // create subpaths property on new object
    if (!nested[index].subpaths) { nested[index].subpaths = [] }

    // add child paths to existing parent
    nested[index].subpaths.push(flat[i])
  }
}

console.log(nested);

I'm also open to using jQuery or Underscore if they expose any functionality that would tidy up the source code. I'd also like to modify the code so it could handle this recursively with an unspecified level of depth on each node.


texts: Avoid iteration on complete list to find (fuzzy) matching string

texts:
I have about 10k categories, represented by their name.
I need to find a match for a category input among this list of 10k categories.

This is done through an API, and by batches: the endpoint will receive about 500 categories to match.

The process is:

-> Receive request with all categories to match

-> For each word, run fuzzy matching algorithm with the 10k categories.

-> Return match

I'm using Fuzzy Wuzzy for the algorithm, and Django for the API.
Basically, this would look like this:

response = {}

for category in categories_received:
  for master_category in master_categories:
    if fuzz.ratio(category, master_category) > 95:
       response[category] = master_category

This is terribly inefficient, but I couldn't find another way to do it.
I control both sides: the way data is sent to the API, and of course the way it is compared to the existing categories.

Any idea/input on how to make this code more efficient would be much appreciated


texts: Retrieving list of flight numbers from flight path

texts:
The below is \$O(n^3)\$. I'm sure there is a way to improve this..

//vector<string> flight_path; given as parameter
//vector<int> flight_number; need to fill
//vector<segment> flight_segments; known, contains segments (dep, arr, flightnum)

//Purpose of the function is to take in a flight_path and give back a list of
// flight numbers for each segment
//
// Input: New York, London, Kolkata
// Output: 101 201 301, 102 939

vector<string> get_flight_numbers (vector<string> flight_path) {

    vector<int> flight_number;

    for (int i = 0; i < flight_path.size() - 1; i++) {

        string dep = flight_path.at(i);
        string arr = flight_path.at(i+1);

        for (int j = 0; j < flight_segments.size(); j ++) {
            if (flight_segments.at(j).departure == dep && flight_segments.at(j).arrival == arr) {
                for (int k = 0; k < flight_segments.at(j).segment_numbers.size(); k++) {
                    flight_numbers.push_back(atoi(flight_segments.at(j).segment_numbers.at(k).c_str()));
                }
            }
        }

    }
}


texts: Maximum consecutive subarray sum for a variable sliding window size

texts:
The Problem:

Given an array of integers L find the largest sum of a consecutive subarray of size k or less.
Constraints:

2 <= len(L) <= 10000

3 <= k <= len(L)

each element in the array will have an absolute value no more than 200

there will be at least one positive integer in the array L

Samples:
L=[-200, 91, 82, 43], k=3, the result should be 216
L=[41, 90, -62, -16, 25, -61, -33, -32, -33], k=5, the result should be 131

The Implementation:
Initially, I started with brute-forcing the problem, probing all sizes of the sliding window starting from k to 1. This immediately got me into "time limit exceeded" situation.
The idea implemented below is based on picking out only positive integers initially. For every positive integer, we are looking if the following integers in a window are contributing to the current sum or not, cutting off situations when current sum drops below zero:
def maximum_consecutive_subarray(L, k):
    global_max = 0

    for index in range(len(L) - 1, -1, -1):
        if L[index] < 0:  # skipping all negative values
            continue

        sliding_index = index
        positive_count = positive_sum = 0

        while sliding_index >= 0 and positive_count < k:
            if L[sliding_index] >= 0:
                positive_count += 1
                positive_sum += L[sliding_index]

                global_max = max(global_max, positive_sum)
            else:
                negative_count = 1
                negative_sum = L[sliding_index]

                while sliding_index - 1 >= 0 > L[sliding_index - 1]:  # iterating over all consecutive negative values
                    negative_count += 1
                    negative_sum += L[sliding_index - 1]
                    sliding_index -= 1

                    if positive_count + negative_count == k:  # break if sliding window size limit reached
                        break

                if positive_sum + negative_sum > 0:  # if we still contribute to the maximum value
                    positive_count += negative_count
                    positive_sum += negative_sum
                else:
                    break  # exit this window if nothing was to contribute

            sliding_index -= 1

    return global_max

I would like to know if we can further improve the solution time complexity wise and would appreciate any other feedback.


texts: Calculate CPU% by process from /proc/stat and /proc//stat

texts:
We don't have the possibility to install modules like psutil on some boxes, thus I decided to write a simple script to calculate the CPU percentage by the given PID.

It uses /proc/stat to get CPU times and /proc/<pid>/stat to get the process CPU usage.

I added top output here, just to compare results and to be sure it's (about) correct. But - I don't like to use external utilities in Python scripts, thus top will be removed.

OS: CentOS 6.5
Python: 2.6.6

Next - it calculates % of CPU, taken by this process:

#!/usr/bin/env python

import sys
import os
import time
import subprocess

if len(sys.argv) == 2:
    pid = sys.argv[1]
else:
    print('No PID specified. Usage: %s <PID>' % os.path.basename(__file__))
    sys.exit(1)

def proct(pid):
    try:
        with open(os.path.join('/proc/', pid, 'stat'), 'r') as pidfile:
            proctimes = pidfile.readline()
            # get utime from /proc/<pid>/stat, 14 item
            utime = proctimes.split(' ')[13]
            # get stime from proc/<pid>/stat, 15 item
            stime = proctimes.split(' ')[14]
            # count total process used time
            proctotal = int(utime) + int(stime)
            return(float(proctotal))
    except IOError as e:
        print('ERROR: %s' % e)
        sys.exit(2)

def cput():
    try:
        with open('/proc/stat', 'r') as procfile:
            cputimes = procfile.readline()
            cputotal = 0
            # count from /proc/stat: user, nice, system, idle, iowait, irc, softirq, steal, guest
            for i in cputimes.split(' ')[2:]:
                i = int(i)
                cputotal = (cputotal + i)
            return(float(cputotal))
    except IOError as e:
        print('ERROR: %s' % e)
        sys.exit(3)

# assign start values before loop them
proctotal = proct(pid)
cputotal = cput()

try:
    while True:

        # for test, to compare results
        proc = subprocess.Popen("top -p %s -b -n 1 | grep -w mysql | awk '{print $9}'" % pid, shell=True, stdout=subprocess.PIPE)
        cpu_percentage = proc.communicate()
        print('With TOP: %s' % (cpu_percentage[0].rstrip('\n')))

        pr_proctotal = proctotal
        pr_cputotal = cputotal

        proctotal = proct(pid)
        cputotal = cput()

        try:
            res = ((proctotal - pr_proctotal) / (cputotal - pr_cputotal) * 100)
            print('With script: %s\n' % round(res, 1))
        except ZeroDivisionError:
            pass

        time.sleep(1)
except KeyboardInterrupt:
    sys.exit(0)

And its results:

$ ./cpu_usage_by_proccess.py 24846
With TOP: 0.0
With script: 0.0

With TOP: 0.0
With script: 0.0

With TOP: 0.0
With script: 0.7

With TOP: 14.0
With script: 4.8

With TOP: 32.0
With script: 34.4

With TOP: 29.9
With script: 32.6

With TOP: 30.0
With script: 35.1

With TOP: 18.0
With script: 26.5

With TOP: 20.0
With script: 20.6

What's done incorrectly here, or what can be done better (maybe a more "Pythonic way")?


texts:
I have a stream of YUV422 encoded images coming in from a camera using Direct Show. I then run this byte array through a conversion method and return the RGB array to C# to display in a WPF app.

My problem is that running the application for a 10 minute period and profiling it, the CPU usage is too high for my usage, and it spends around 70-80% of the time in the YUV422 to RGB conversion method.

The method works really well but is just too slow.

Are there any optimizations I am missing here? (I am mainly a C# programmer so I can easily miss C++ optimizations)

*nb this method is in a C++ dll that is "PInvoked" from C# - the reason for C++ is that a similar method takes twice as long in C# than C++

const int STANDARD_SIZE = 1036800;

// Clamp out of range values
#define CLAMP(t) (((t)>255)?255:(((t)<0)?0:(t)))

// Color space conversion for RGB
#define GET_R_FROM_YUV(y, u, v) ((298*y+409*v+128)>>8)
#define GET_G_FROM_YUV(y, u, v) ((298*y-100*u-208*v+128)>>8)
#define GET_B_FROM_YUV(y, u, v) ((298*y+516*u+128)>>8)

BOOL __stdcall YUV422toRGB888(unsigned char *d, unsigned char *s)
{
    for (unsigned int i = 0; i < STANDARD_SIZE; ++i)
    {
        int y0 = *s++ - 16;
        int u0 = *s++ - 128;
        int y2 = *s++ - 16;
        int v = *s++ - 128;

        // BGR
        *d++ = CLAMP(GET_B_FROM_YUV(y0, u0, v));
        *d++ = CLAMP(GET_G_FROM_YUV(y0, u0, v));
        *d++ = CLAMP(GET_R_FROM_YUV(y0, u0, v));

        // BGR
        *d++ = CLAMP(GET_B_FROM_YUV(y2, u0, v));
        *d++ = CLAMP(GET_G_FROM_YUV(y2, u0, v));
        *d++ = CLAMP(GET_R_FROM_YUV(y2, u0, v));
    }

    return true;
}